20250306INT to Staff 8 Attachment.pdf

DOE/EE-0829
Y
O
3
W
z
z
O
U Energy Efficiency Program
U u Impact Evaluation Guide
U pW
v Evaluation, Measurement, and Verification Working Group
LL
u_
W
December 2012
W
Ujz
W
J
Q
UjU
0
(n LU
0
. o
0
Qp 9
W�
I
o
o
The State and Local Energy Efficiency Action
Network is a state and local effort facilitated by the
federal government that helps states, utilities, and
other local stakeholders take energy efficiency to
scale and achieve all cost-effective energy
efficiency by 2020.
Learn more at www.seeaction.energy.gov

ABOUT THIS DOCUMENT
This Energy Efficiency Program Impact Evaluation Guide describes the common terminology,structures,and approaches used
for determining(evaluating)energy and demand savings as well as avoided emissions and other non-energy benefits resulting
from facility(non-transportation)energy efficiency programs that are implemented by local governments,states,utilities,private
companies,and nonprofits.While this guide does not recommend specific approaches,it provides context,planning guidance,
and discussion of issues that determine the most appropriate evaluation objectives and best practices approaches for different
efficiency portfolios. By using standard evaluation terminology and structures and best practices approaches,evaluations can
support the adoption,continuation,and expansion of effective efficiency actions.
The primary audiences for this guide are energy regulators;public and private energy efficiency portfolio administrators such
as utilities,nonprofit organizations,and government agencies; program implementers;and evaluators looking for guidance on
the following:
• The evaluation process and approaches for determining program impacts
• Planning evaluation efforts
• Key issues associated with establishing evaluation frameworks for improving the efficacy of energy efficiency portfolios,
documenting the impacts of such portfolios,and comparing demand-and supply-side resources.
Introductory portions and appendices are also intended for policymakers seeking general information about efficiency program
impact evaluation as well as the basic principles of process and market evaluations and cost-effectiveness analyses.Although the
guide is not directly intended for expert evaluation practitioners who can rely on more detailed and specific resources that are
referred to in this guide,it offers introductions to and summaries of evaluation topics that can be useful for explaining concepts
and standard practices to clients, new staff,stakeholders,and others who could benefit from a refresher on principles.
This 2012 version of the guide is an update to the 2007 National Action Plan for Energy Efficiency Model Energy Efficiency Program
Impact Evaluation Guide. Prepared by Steve Schiller,Schiller Consulting,Inc.www.epa.gov/eeactionplan.
www.seeaction.energy.gov December 2012

The Energy Efficiency Program Impact Evaluation Guide was developed as a product of the State and Local Energy Efficiency(EPA)Action
Network(SEE Action),which is facilitated by the U.S. Department of Energy and the U.S.Environmental Protection Agency.Content does not
imply an endorsement by the individuals or organizations that are part of SEE Action working groups or reflect the views,policies,or other-
wise of the federal government.
This effort was funded by the Permitting,Siting,and Analysis Division of the U.S. Department of Energy's Office of Electricity
Delivery and Energy Reliability under Lawrence Berkeley National Laboratory Contract No. DE-ACO2-05CH11231.
If this document is referenced,it should be cited as:
State and Local Energy Efficiency Action Network.2012.Energy Efficiency Program Impact Evaluation Guide.
Prepared by Steven R.Schiller,Schiller Consulting, Inc.,www.seeaction.energy.gov.
FOR MORE INFORMATION
For more information regarding the Energy Efficiency Program Impact Evaluation Guide,please contact:
Michael Li Carla Frisch
U.S. Department of Energy U.S. Department of Energy
Email: [e-mail] Email:[e-mail]
Regarding the State and Local Energy Efficiency Action Network,please contact:
Johanna Zetterberg
U.S. Department of Energy
Email:[e-mail]
December 2012 www.seeaction.energy.gov

Acknowledgments
The following people provided key input to the development of this version of the guide by providing materials,sidebars,and/or extremely
valuable review and input on draft versions(*denotes member of SEE Action's EM&V Working Group):
• Jeffrey Brown, Robyn DeYoung,and Nikolaas Dietsch,U.S. Environmental Protection Agency
• Tom Eckman, Northwest Power Planning Council
• Hilary Forster,Consortium for Energy Efficiency*
• Donald Gilligan,National Association of Energy Services Companies*
• Fred Gordon, Energy Trust of Oregon*
• Dennis Hartline, Maryland Energy Administration*
• M.Sami Khawaja, Hossein Haeri,Tina Jayaweera,and David Sumi,The Cadmus Group
• Stephen Kromer, Kromer Engineering
• Jennifer Meissner, New York State Energy Research and Development Authority*
• Mike Messenger, Itron
• Julie Michals*and Elizabeth Titus, Northeast Energy Efficiency Partnerships
• Peter Miller, Natural Resources Defense Council*
• Jane Peters, Research Into Action
• Mitch Rosenberg, DNV KEMA
• Lisa Skumatz,Skumatz Economic Research Associates
• Rodney Sobin,Alliance to Save Energy*
• Annika Todd, Lawrence Berkeley National Laboratory
• Carol Zabin, University of California, Berkeley.
Dr. Khawaja provided substantial input into several sections of this guide,as did the EPA staff on the Avoided Emissions Chapter;their input is
therefore particularly acknowledged.
Mark Wilson provided editing services. Publication and graphics were provided by the National Renewable Energy Laboratory.
v www.seeaction.energy.gov December 2012

List of Acronyms
EERS:energy efficiency resource standard
ACEEE:American Council for an Energy-Efficient Economy EGU:electric generating unit(a power plant)
AEA:American Evaluation Association EM&V:evaluation,measurement,and verification
ANSI:American National Standards Institute EPA:U.S.Environmental Protection Agency
ASHRAE:American Society of Heating, Refrigerating and ER:emission rate
Air-Conditioning Engineers ESCO:energy services company
ETD:Energy Trust of Oregon
EUL:effective useful life
BAU:business as usual
BM:build margin(for electric generating units)
Btu:British thermal units FEMP: Federal Energy Management Program
FERC:Federal Energy Regulatory Commission
CAIR:Clean Air Interstate Rule
CALMAC:California Measurement Advisory Council GHG:greenhouse gas
C&S:(efficiency)codes and standards
CDD:cooling degree day
CDM:(United Nations Framework Convention on Climate Change) HDD:heating degree day
Clean Development Mechanism HERS:Home Energy Rating System
CEMS:Continuous Emission Monitoring Systems HHV:higher heating value
CFL:compact fluorescent light bulb HVAC:heating,ventilation,and air conditioning
CMVP:Certified Measurement and Verification Professionals
COz:carbon dioxide
CPUC:California Public Utility Commission IPCC:Intergovernmental Panel on Climate Change
CSA:conditional savings analysis IPM:integrated planning model
CV:contingent valuation IPMVP:International Performance Measurement and
Verification Protocol
Cx:commissioning
IRP:integrated resource planning
ISO:independent system operator or International Organization
for Standardization
DEER: Database for Energy Efficiency Resources(California)
DOE: U.S.Department of Energy
DR:demand response kW:kilowatt
DSM:demand-side management kWh: kilowatt-hour
E&T:education and training lb: pound
ECM:energy conservation measure
EE:energy efficiency
December 2012 www.seeaction.energy.gov v

M&V:measurement and verification RASS:residential appliance saturation studies
MARAMA:Mid-Atlantic Regional Air Management Association RCx: retro-commissioning
MMBtu:million Btu RCT:randomized controlled trial
MT:market transformation RD&D:research,development,and demonstration
MW:megawatt REED: Regional Energy Efficiency Database
MWh:megawatt-hour RFP:request for proposal
RFQ: request for qualifications
RGGI: Regional Greenhouse Gas Initiative
NAESCO: National Association of Energy Service Companies RIM:ratepayer impact test
NEB:non-energy benefit RTF:Regional Technical Forum
NEEA: Northwest Energy Efficiency Alliance
NEI:non-energy impact S
NERC: North American Reliability Corporation SCT:societal cost test
NOMAD: naturally occurring market adoption rate SIP:State(air pollution reduction)Implementation Plan
NOx:nitrogen oxide SOz:sulfur dioxide
NPV:net present value SPM:(California)Standard Practice Manual
NTG:net-to-gross SPT:standardized project tracking
NTGR:net-to-gross ratio
TBE:theory-based evaluation
O&M:operations and maintenance T&D:transmission and distribution
OM:operating margin(for electric generating units) TRC:total resource cost test
TRM:technical reference manual
PACT:program administrator cost test
PCT:participant cost test UMP:Uniform Methods Project
PMP:Performance Measurement Protocols for Commercial
Buildings(ASHRAE)
PSC:Public Service Commission VCS:Verified Carbon Standard
PUC: Public Utilities Commission
WBCSD:World Business Council for Sustainable Development
QAG:quality assurance guideline WRI:World Resources Institute
QA/QC:quality assurance/quality control
QEM:quasi-experimental methods
vi www.seeaction.energy.gov December 2012

Table of Contents
Acknowledgments........................................................................................................................................................................................................iv
Listof Acronyms.............................................................................................................................................................................................................v
Listof Tables.................................................................................................................................................................................................................. A
Listof Figures............................................................................................................................................................................................................... xii
ExecutiveSummary.........................................................................................................................................................................................xiii
ES.1 Topics Covered by this Guide and Intended Audiences.............................................................................................................................xiii
ES.2 Definition and Importance of Energy Efficiency Evaluation.......................................................................................................................xiii
ES.3 Impact Evaluation Metrics.......................................................................................................................................................................... xiv
ES.4 Energy Savings Estimates and Uncertainty:"How Good is Good Enough?"..............................................................................................xv
ES.S Evaluation and the Efficiency Program Process...........................................................................................................................................xv
ES.5.1 Planning Impact Evaluations............................................................................................................................................................ xvi
ES.5.2 Implementing Impact Evaluations................................................................................................................................................... xvi
ES.5.3 Evaluation Planning Characteristics and Frameworks.....................................................................................................................xvii
ES.5.4 Evaluation Planning Issues..............................................................................................................................................................xviii
Chapter1 Introduction....................................................................................................................................................................................1-1
1.1 Guide Objective............................................................................................................................................................................................1-1
1.2 Subjects Covered in This Guide....................................................................................................................................................................1-1
1.3 Guide Structure and How to Use This Guide...............................................................................................................................................1-2
1.4 Source Documents........................................................................................................................................................................................1-2
Chapter 2 Energy Efficiency Program Evaluation Overview.............................................................................................................................2-1
2.1 Energy Efficiency Program Categories and Definitions................................................................................................................................2-1
2.1.1 Policy Context of Energy Efficiency Programs....................................................................................................................................2-1
2.1.2 Efficiency Program Categories...........................................................................................................................................................2-1
2.1.3 Savings Hierarchy...............................................................................................................................................................................2-2
2.2 Program Evaluation Categories and Impact Evaluation Definitions............................................................................................................2-3
2.2.1 Evaluation Definitions........................................................................................................................................................................2-4
2.3 Objectives and Importance of Evaluation....................................................................................................................................................2-5
Chapter3 Impact Evaluation Basics.................................................................................................................................................................3-1
3.1 Impact Evaluation Process............................................................................................................................................................................3-1
3.1.1 Verification and Documentation........................................................................................................................................................3-1
3.1.2 The Evaluator's Role and Reporting of Savings..................................................................................................................................3-2
3.2 Energy and Non-Energy Benefit Evaluation Metrics....................................................................................................................................3-3
December 2012 www.seeaction.energy.gov vii

3.3 Fundamental Energy Efficiency Impact Evaluation Concepts:The Counterfactual and Managing Uncertainty........................................3-4
3.3.1 The Counterfactual.............................................................................................................................................................................3-4
3.3.2 Baselines.............................................................................................................................................................................................3-4
3.4 Determining Energy and Demand Savings...................................................................................................................................................3-5
3.4.1 Noncontrol Group Impact Evaluation Approaches............................................................................................................................3-5
3.4.2 Control Group Approaches:Large-Scale Consumption Data Analysis
(Randomized Controlled Trials and Quasi-Experimental Methods)..................................................................................................3-6
3.4.3 Approaches for Determining Net Energy and Demand Savings........................................................................................................3-7
Chapter4 Calculating Energy Savings..............................................................................................................................................................4-1
4.1 Measurement and Verification Approach....................................................................................................................................................4-1
4.1.1 Measurement and Verification Approaches......................................................................................................................................4-3
4.1.2 M&V Option A: Retrofit Isolation—Key Parameter Measurement...................................................................................................4-5
4.1.3 M&V Option B:Retrofit Isolation—All Parameter Measurement.....................................................................................................4-5
4.1.4 M&V Option C:Whole-Facility Analyses............................................................................................................................................4-6
4.1.5 M&V Option D:Calibrated Simulation...............................................................................................................................................4-6
4.2 Deemed Savings Approach...........................................................................................................................................................................4-7
4.3 Large-Scale Consumption Data Analysis(Randomized Controlled Trials and Quasi-Experimental Methods)............................................4-9
4.3.1 Randomized Controlled Trial(RCT) Methods.....................................................................................................................................4-9
4.3.2 Quasi-Experimental Methods(QEMs).............................................................................................................................................4-10
4.3.3 Analysis Techniques..........................................................................................................................................................................4-11
4.4 Selecting an Energy Savings Evaluation Approach.....................................................................................................................................4-11
4.4.1 M&V Approach.................................................................................................................................................................................4-12
4.4.2 Deemed Savings Approach..............................................................................................................................................................4-12
4.4.3 Large-Scale Consumption Data Analysis Approach.........................................................................................................................4-13
Chapter5 Determining Net Energy Savings.....................................................................................................................................................5-1
5.1 Definitions and Uses of Net Savings.............................................................................................................................................................5-1
5.1.1 Defining Net Savings..........................................................................................................................................................................5-1
5.1.2 Importance and Uses of Net Savings.................................................................................................................................................5-3
5.2 Approaches for Determining Net Savings....................................................................................................................................................5-3
5.2.1 Self-Reporting Surveys.......................................................................................................................................................................5-5
5.2.2 Enhanced Self-Reporting Surveys......................................................................................................................................................5-6
5.2.3 Stipulated Net-to-Gross Ratio............................................................................................................................................................5-7
5.2.4 Selecting a Net Savings Evaluation Approach and Timing.................................................................................................................5-7
5.3 Issues and Nuances of Net Savings..............................................................................................................................................................5-7
5.3.1 Free Ridership....................................................................................................................................................................................5-8
5.3.2 Spillover and Market Effects..............................................................................................................................................................5-8
5.3.3 Baselines and Net Savings..................................................................................................................................................................5-9
5.4 Summary.......................................................................................................................................................................................................5-9
viii www.seeaction.energy.gov December 2012

Chapter 6 Calculating Avoided Air Emissions...................................................................................................................................................6-1
6.1 Energy Efficiency and Avoided Emissions Summary....................................................................................................................................6-1
6.2 Special Considerations for Calculating Avoided Emissions from Efficiency Programs.................................................................................6-2
6.2.1 Additionality.......................................................................................................................................................................................6-2
6.2.2 Assessment Boundary Issues: Primary and Secondary Effects/Direct and Indirect Emissions.........................................................6-3
6.2.3 Special Issues for Capped Pollutants under Cap-and-Trade Programs..............................................................................................6-4
6.2.4 Avoided Emissions Calculations for Different Objectives..................................................................................................................6-5
6.3 Direct Onsite Avoided Emissions Calculations.............................................................................................................................................6-5
6.4 Emission Factors for Grid-Connected Electric Generating Units...................................................................................................................6-6
6.4.1 The Electricity Generation Mix...........................................................................................................................................................6-6
6.4.2 Using Avoided Emission Factors and Scenario Analyses for Electric Grid Analyses..........................................................................6-8
6.5 Selecting an Approach for Calculating Avoided Emissions........................................................................................................................6-14
6.6 Additional Resources on Avoided Emissions Calculations.........................................................................................................................6-14
Chapter7 Impact Evaluation Considerations...................................................................................................................................................7-1
7.1 Selecting Baselines.......................................................................................................................................................................................7-1
7.1.1 Existing Conditions Baselines.............................................................................................................................................................7-1
7.1.2 Codes and Standards Baseline...........................................................................................................................................................7-2
7.1.3 Common Practice Baseline................................................................................................................................................................7-2
7.1.4 Defining Baselines for Specific Program Types..................................................................................................................................7-2
7.2 Determining Demand Savings......................................................................................................................................................................7-4
7.3 Persistence of Savings...................................................................................................................................................................................7-6
7.3.1 Definitions..........................................................................................................................................................................................7-6
7.3.2 Rebound.............................................................................................................................................................................................7-7
7.3.3 Determining Persistence....................................................................................................................................................................7-7
7.4 Controlling Uncertainty................................................................................................................................................................................7-8
7.4.1 Systematic Errors/Bias........................................................................................................................................................................7-9
7.4.2 Random Errors...................................................................................................................................................................................7-9
7.5 Evaluation Budgets: Balancing the Value of Evaluation Results and Uncertainty.....................................................................................7-11
7.5.1 Using Impact Evaluations to Manage Risk.......................................................................................................................................7-11
7.5.2 Budget-Setting Considerations.........................................................................................................................................................7-12
7.5.3 State Evaluation Budgets..................................................................................................................................................................7-14
7.6 Evaluation Principles...................................................................................................................................................................................7-14
7.7 Using Impact Evaluations for Program Feedback.......................................................................................................................................7-18
7.8 Evaluation for Demand Forecasting and Energy Resource Planning.........................................................................................................7-19
7.9 Determining Non-Energy Benefits.............................................................................................................................................................7-20
7.10 Impact Evaluation for Unique Energy Efficiency Program Types.............................................................................................................7-24
7.10.1 Residential Behavior-Based Programs...........................................................................................................................................7-24
7.10.2 Education and Training(E&T) Programs........................................................................................................................................7-25
December 2012 www.seeaction.energy.gov x

7.10.3 Market Transformation(MT)Programs.........................................................................................................................................7-25
7.10.4 Codes and Standards Programs.....................................................................................................................................................7-26
7.10.5 Demand Response(DR)Programs.................................................................................................................................................7-28
7.10.6 Greenhouse Gas(GHG)Mitigation Programs................................................................................................................................7-28
Chapter8 Impact Evaluation Planning.............................................................................................................................................................8-1
8.1 Integration of Evaluation into the Portfolio Implementation Cycle.............................................................................................................8-1
8.2 Hierarchy of Evaluation Planning and Reporting Documents......................................................................................................................8-3
8.2.1 Evaluation Planning Documents........................................................................................................................................................8-3
8.2.2 Evaluation Reports.............................................................................................................................................................................8-4
8.3 Principles and Issues That Determine the Scope of an Impact Evaluation..................................................................................................8-5
8.3.1 Defining Evaluation Objectives and Metrics......................................................................................................................................8-6
8.3.2 Evaluator Roles and Selection............................................................................................................................................................8-8
8.3.3 Setting the Boundary and Granularity of Reported Results............................................................................................................8-10
8.3.4 Schedule for Evaluation and Reporting............................................................................................................................................8-11
8.3.5 How Are Evaluated Savings Estimates Applied—Looking Back Or Going Forward?.......................................................................8-11
8.3.6 Data Management Strategies..........................................................................................................................................................8-12
8.3.7 Addressing Disputes.........................................................................................................................................................................8-13
8.4 Portfolio Cycle Impact Evaluation and M&V Plan Outlines........................................................................................................................8-13
8.4.1 Portfolio Cycle Impact Evaluation Plan and Report Outlines...........................................................................................................8-13
8.4.2 Project-Specific M&V Plan and Report Outlines..............................................................................................................................8-15
8.4.3 Checklist of Planning Decisions for an Impact Evaluation...............................................................................................................8-15
AppendixA:Glossary.....................................................................................................................................................................................A-1
Appendix B:Other Evaluation Categories and Approaches............................................................................................................................B-1
B.1 Process,Market Effects,and Cost-Effectiveness Evaluations...................................................................................................................... B-1
B.1.1 Process Evaluations........................................................................................................................................................................... B-1
B.1.2 Market Effects Evaluations................................................................................................................................................................ B-3
B.1.3 Cost-Effectiveness Analyses.............................................................................................................................................................. B-4
B.2 Top-Down Impact Evaluation...................................................................................................................................................................... B-9
AppendixC:Resources...................................................................................................................................................................................C-1
CABackground...................................................................................................................................................................................................C-1
C.2 Primary Impact Evaluation Resources..........................................................................................................................................................C-1
C.2.1 General Efficiency Evaluation Resource Websites.............................................................................................................................C-1
C.2.2 Select Impact Evaluation Resources..................................................................................................................................................C-1
C.2.3 U.S.Environmental Protection Agency Non-Energy Benefits and Avoided Emissions Calculation References...............................C-2
C.2.4Technical Reference Manual(TRMs)Resources................................................................................................................................C-3
C.3 Measurement and Verification Resources...................................................................................................................................................C-5
References.....................................................................................................................................................................................................D-1
x www.seeaction.energy.gov December 2012

List of Tables
Table 1.1:Summary of Guide Content and Intended Audience for Each Part of the Guide.............................................................................1-3
Table 2.1:Summary of Evaluation Categories and Types..................................................................................................................................2-4
Table4.1:IPMVP Options Summary..................................................................................................................................................................4-4
Table 4.2:Applications for Each IPMVP M&V Option......................................................................................................................................4-14
Table 4.3:Factors Affecting the Cost of Each M&V Option..............................................................................................................................4-14
Table 4.4:Key Project Characteristics and Applicability for Different M&V Options.......................................................................................4-15
Table 5.1:Audiences for Net Energy Savings.....................................................................................................................................................5-4
Table 5.2:Example of Assigning Free Ridership Percentages............................................................................................................................5-6
Table 6.1:Choosing an Avoided Grid Electricity Emissions Quantification Approach.......................................................................................6-9
Table 7.1:Standard Practices for Selection of Baselines for Common Program Categories.............................................................................7-3
Table 7.2:Electric and Gas EM&V Expenditure and Budget Dollars,United States and Canada(millions USD)*..........................................7-15
Table 7.3:Ratepayer-Funded Energy Efficiency Budgets and EM&V Expenditures for Selected States.........................................................7-16
Table 7.4:EM&V 2008 Funding Levels and Allocation Among Activities........................................................................................................7-17
Table 8.1:Maryland Public Service Commission EM&V,Data Tracking,and Reporting Act Roles
and Responsibilities for Utilities and Statewide Evaluation Consultants........................................................................................8-16
Table 8.2:Energy Efficiency M&V Plan Contents:General Components.........................................................................................................8-20
Table 8.3:Energy Efficiency Project-Specific and Measure-Specific M&V Plan Contents...............................................................................8-21
Table 8.4:Evaluation Planning Checklist .........................................................................................................................................................8-22
TableB.1:Program Evaluation Types................................................................................................................................................................. B-1
TableB.2:Elements of Typical Process Evaluations.......................................................................................................................................... B-2
Table B.3:Approaches for Assessing Market Effects,Including Attribution .................................................................................................... B-5
Table B.4:The Five Principal Cost-Effectiveness Tests Used in Energy Efficiency............................................................................................. B-7
Table B.S:Description of Benefits and Costs Included in Each Cost-Effectiveness Test.................................................................................... B-8
Table B.6:Summary of Benefits and Costs Included in Each Cost-Effectiveness Test....................................................................................... B-9
Table CA:United States Technical Reference Manuals......................................................................................................................................C-3
December 2012 www.seeaction.energy.gov xi

List of Figures
FigureESA:Evaluation objectives ..................................................................................................................................................................... xiv
Figure ES.2:Workflow and reporting for planning,implementing,and evaluating efficiency programs .........................................................xv
Figure ES.3:Evaluation activities workflow........................................................................................................................................................ xvi
Figure ESA:Evaluation is integral to a typical cyclic planning-implementation-evaluation process...............................................................xviii
Figure ES.5:Hierarchy of EM&V planning documents........................................................................................................................................xix
Figure 2.1:Hierarchy of energy efficiency activities..........................................................................................................................................2-2
Figure 2.2:Continuum of energy efficiency actions...........................................................................................................................................2-6
Figure 3.1:True program savings:the counterfactual for a residential household energy efficiency program...............................................3-4
Figure 3.2:Energy consumption before,during,and after a project is implemented......................................................................................3-5
Figure 4.1:M&V option selection flowchart....................................................................................................................................................4-16
Figure 5.1:Single program evaluation components..........................................................................................................................................5-2
Figure 5.2:Multiple program evaluation overlaps.............................................................................................................................................5-2
Figure 6.1:Example electricity load duration curve...........................................................................................................................................6-7
Figure6.2:U.S.EPA eGrid subregions..............................................................................................................................................................6-10
Figure 7.1: Incremental value of information versus incremental cost of evaluation ....................................................................................7-13
Figure 7.2:Components of C&S savings estimate............................................................................................................................................7-28
Figure 8.1:Program implementation cycle with high-level evaluation activities..............................................................................................8-2
Figure B.1:Comparison of bottom-up versus top-down evaluation.............................................................................................................. B-10
Figure B.2:Graphical illustration of estimation of market gross savings........................................................................................................ B-10
xii www.seeaction.energy.gov December 2012

Executive Summary
ES.1 TOPICS COVERED BY THIS GUIDE Documents from the U.S.Department of Energy's(DOE'S)Uniform
AND INTENDED AUDIENCES Methods Project(UMP)serve as a companion set to this guide and
This Energy Efficiency Program Impact Evaluation Guide describes and include model evaluation plans for specific energy efficiency measures
provides guidance on approaches for determining and documenting and program categories(e.g.,residential lighting,refrigerators,
energy and non-energy benefits resulting from end-use energy commercial cooling).'
efficiency programs and portfolios of programs. It specifically focuses ES.2 DEFINITION AND IMPORTANCE OF
on impact evaluations for programs designed to reduce facility(e.g., ENERGY EFFICIENCY EVALUATION
home,commercial building,factory)energy consumption and/or
demand as well as related air emissions.This guide's objective is to Energy efficiency evaluation includes any of a range of assessment
support the implementation of effective energy efficiency actions by studies and other activities aimed at determining the effects of
providing information on standard procedures and best practices for an energy efficiency program. Evaluations can document program
planning and conducting evaluations and reporting results.To this performance,operations,changes in energy efficiency markets,and
end,this guide accomplishes the following: cost-effectiveness.There are three broad categories of efficiency
• Defines a systematic evaluation planning and program evaluations:impact evaluations,process evaluations,and
implementation process
market evaluations.Although this guide focuses on impact evalua-
tions,it is helpful to know the purposes and goals of all three:
• Describes several standard approaches for determining
• Impact evaluations:assessments that determine and document
energy and demand savings(as well as avoided emissions
the direct and indirect benefits of an energy efficiency program.
and other non-energy impacts)
Impact evaluation involves real-time and/or retrospective
• Defines key terms related to energy efficiency evaluation assessments of the performance and implementation of an
• Provides guidance on key evaluation issues efficiency program or portfolio of programs.Program benefits,
• Lists publicly available energy efficiency evaluation resources. or impacts,can include energy and demand savings and non-
energy benefits(sometimes called co-benefits,with examples
The programs primarily addressed in this guide are voluntary;that being avoided emissions,health benefits,job creation and local
is,program participants choose to take the efficiency actions as a economic development,energy security,transmission and
result of some form of inducement.This guide does not focus on, distribution benefits,and water savings).Impact evaluations also
but does touch on,evaluating mandatory requirements for efficiency support cost-effectiveness analyses aimed at identifying relative
such as found in codes and standards.Similarly,the guide only briefly program costs and benefits of energy efficiency as compared to other
addresses evaluating programs for which energy savings are an energy resources,including both demand-and supply-side options.
indirect benefit,such as contractor training programs. Process evaluations:formative,systematic assessments of an
energy efficiency program.They document program operations
The audiences for this guide are program designers,implementers, and identify and recommend improvements that are likely to
administrators,evaluators,and public agency officials who oversee
increase the program's efficiency or effectiveness for acquiring
and implement energy efficiency programs. Introductory portions of energy efficiency resources, preferably while maintaining high
this guide are intended for policymakers seeking information about
the basic principles of impact evaluation.Those looking for just the levels of participant satisfaction.
basics may want to read only through Chapter 3 and refer to the Market evaluations:assessments of structure or functioning of
appendices for overviews of other evaluation types,definitions,and a market,the behavior of market participants,and/or market
references.Some readers who are new to evaluation assignments changes that result from one or more program efforts. Market
can benefit from reading the entire document,while others may evaluation studies may include estimates of the current market
benefit from focusing on the evaluation issues and planning chapters role of energy efficiency(market baselines),as well as the
(Chapters 7 and 8,respectively)and using the rest of the document potential role of efficiency in a local,state, regional,or national
as a reference.Although the guide is not intended for expert evalu- market(potential studies). Market evaluation studies indicate
ation practitioners,they may find it useful for explaining evaluation how the overall supply chain and market for energy efficiency prod-
concepts to those without their expertise. ucts works and how they have been affected by a program(s).
December 2012 www.seeaction.energy.gov xiii

Market evaluations are critical for,but not exclusively used for, Many energy efficiency evaluations are oriented toward developing
programs with market transformation elements and objectives. retrospective estimates of energy savings attributable to a program
Examples of market evaluations are potential studies,baselines to demonstrate in regulatory proceedings that public or energy
studies,and market effects studies. consumer funds were properly and effectively spent. Beyond
documenting savings and attribution,though,is the role of evaluation
Evaluations have three primary objectives,as shown in Figure ESA: in improving programs and providing a basis for future savings
• Document the benefits(i.e.,impacts)of a program and determine estimates in resource plans.Therefore,evaluation both fosters
whether the subject program(or portfolio of programs)met its goals more effective programs and justifies increased levels of investment
• Identify ways to improve current and future programs through in energy efficiency as a long-term,reliable energy resource. Perhaps
determining why program-induced impacts occurred the imperative for conducting evaluation is best described by a quote
• Support energy demand forecasting and resource planning by attributed to John Kenneth Galbraith:"Things that are measured
understanding the historical and future resource contributions tend to improve:'z
of energy efficiency as compared to other energy resources. ES.3 IMPACT EVALUATION METRICS
One or more of the following three metrics are usually reported as
FIGURE ESA: Evaluation objectives the output of impact evaluations:
• Estimates of gross(energy and/or demand)savings.These are
the changes in energy consumption and/or demand that result
directly from program-related actions taken by participants in
an efficiency program,regardless of why they participated.
• Estimates of net(energy and/or demand)savings.These
are the changes in energy consumption or demand that are
attributable to an energy efficiency program.The primary,but
UNDERSTANDnot exclusive,considerations that account for the difference
DOCUMENT • - • between net and gross savings are free riders(i.e.,those who
IMPACTS PROGRAM I would have implemented the same or similar efficiency projects,
PERFORMANCE
to one degree or another,without the program now or in the
near future)and participant and non-participant spillover
(i.e.,savings that result from actions taken as a result of a
SUPPORTIENERGY programs's influence but which are not directly subsidized or
RESOURCErequired by the program).Net savings may also include consider-
ation of market effects(changes in the structure of a market).
Determining net savings involves separating out the impacts
that are a result of influences other than the program being
evaluated,such as consumer self-motivation or effects of
prior and/or other programs.Given the range of influences on
consumers'energy consumption and the complexity in separat-
EVALUATION SUPPORTS SUCCESSFUL ing out both short-term and long-term market effects caused by
EFFICIENCY PROGRAMS the subject programs(and other programs),attributing changes
Documenting the benefits of efficiency,using credible and to one cause(i.e.,a particular program)or another can be quite
transparent methods,is a key component of successfully complex.This is compounded by a general lack of consensus
implementing and expanding the role of efficiency in providing among policymakers and regulators on which short-term and
secure,stable,reliable,clean,and reasonably priced energy. long-term market influences and effects should be considered
Therefore,evaluation is not an end unto itself but an effective when determining net savings. Net savings are discussed in
tool for supporting the adoption,continuation,and expansion of Chapter 5.
energy efficiency programs,and thus the efficient use of energy.
xiv www.seeaction.energy.gov December 2012

• Estimates of non-energy benefits(NEBs).These are the impacts Two principles are important when considering"how good is good
associated with program implementation or participation enough":(1)energy efficiency investments should be cost effective,
aside from energy and demand savings.These results can be and(2)evaluation investments should consider risk management
positive or negative.Some examples include reduced emissions principles and thus balance the costs of evaluation against the value
and environmental benefits,productivity improvements,jobs of the information derived from evaluation(i.e.,evaluation should
created and local economic development, reduced utility also be cost effective).The value of the information is directly related
customer disconnects,greater comfort for building occupants, to the risks of underestimating or overestimating the benefits(savings)
lower maintenance costs due to better equipment,or increased and costs associated with efficiency investments.These risks might
maintenance costs due to new and more complex systems. be associated with errors of commission or errors of omission.An
NEBs are discussed in Section 7.9. error of commission might be overestimating savings,which in turn
can result in continuing programs that are not cost effective and/or
ESA ENERGY SAVINGS ESTIMATES overpaying contractors,administrators,and participants.An error of
AND UNCERTAINTY: "HOW GOOD IS omission,on the other hand,might be associated with underestimating
GOOD ENOUGH?" savings or not implementing efficiency actions because of the difficulty
Each of the bullets in Section ES.3 above defines an"estimate" in documenting savings,both of which can result in underinvesting in
versus an exact value.This is because energy and demand savings as efficiency and relying on other energy resources that have their own
well as non-energy benefits resulting from efficiency actions cannot risks and uncertainties,such as fuel costs and environmental impacts.
be directly measured. Instead,savings and benefits are based on
counterfactual assumptions.Using counterfactual assumptions ES.5 EVALUATION AND THE EFFICIENCY
implies that savings are estimated to varying degrees of accuracy PROGRAM PROCESS
by comparing the situation (e.g.,energy consumption) after a As shown in Figure ES.2,the efficiency program process consists of
program is implemented(the reporting period)to what is assumed planning,implementing,and evaluating activities.Throughout this
to have been the situation in the absence of the program(the process,savings values are typically indicated based on estimates
"counterfactual"scenario,known as the baseline).For energy impacts, prepared as part of each activity.One way to describe these savings
the baseline and reporting period energy use are compared,while is with the following classifications,also displayed in Figure ES.2:
controlling(making adjustments)for factors unrelated to energy Projected savings:values reported by a program implementer
efficiency actions,such as weather or building occupancy.These or administrator before the efficiency activities are completed
adjustments are a major part of the evaluation process; how they
are determined can vary from one program type to another and
from one evaluation approach to another. FIGURE ES.2: Workflow and reporting for planning,
Because the indicated values are estimates,their use as a basis implementing, and evaluating efficiency programs
for decision making can be challenged if their sources and level
of accuracy are not described.Therefore,evaluation results, like
any estimate,should be reported as"expected values";that is, PLAN PROJECTED
based on the impact evaluation,values are expected to be correct PROGRAMS
within an associated level of certainty. Minimizing uncertainty and
balancing evaluation costs with the value of the indicated evaluation Ilk
information are at the heart of the evaluation process and leads to
perhaps the most fundamental evaluation question:"How good is IMPLEMENT CLAIMED
good enough?"This question is a short version of asking(1)what PF PROGRAMS
level of certainty is required for energy savings estimates resulting
from evaluation activities,and(2)is that level of certainty properly Ilk
balanced against the amount of effort(e.g., resources,time,money)
used to obtain that level of certainty? EVALUATE EVALUATED
PROGRAMS SAVINGS
December 2012 www.seeaction.energy.gov xv

• Claimed savings:values reported by a program implementer or ES.5.2 Implementing Impact Evaluations
administrator after the efficiency activities have been completed The impact evaluation is conducted through the following steps:
• Evaluated savings:values reported by an independent third- 1. Verify actual implementation of the program,for example,
party evaluator after the efficiency activities and impact evaluation by confirming installation and proper operation of the energy
have been completed.The designation of"independent"and efficiency measures.This usually also includes auditing and
"third-party" is determined by those entities involved in the validating assumptions used in the program planning process
use of the evaluations and may include evaluators retained,for and checking program tracking databases,project applications,
example, by the program administrator or a regulator. and other documentation and related data records for accurate
With respect to the evaluation activities,they can also be described recording of information.
as consisting of three phases:planning,implementation,and report- 2• Determine first-year program energy(and demand)savings
ing,as shown in Figure ES.3 and described in the next subsections. using one of the following approaches(which are further
defined and described in Chapters 3 and 4):
ES.5.1 Planning Impact Evaluations a. Measurement and verification(M&V):a project-by-project
The following provide the basic steps in planning impact evaluations: approach involving estimating energy and/or demand savings by
1. Define the evaluation objectives and metrics in the context of determining the savings for a representative sample of projects
the evaluated program's(or portfolio's)intended benefits,risks, and applying these projects'savings to the entire population
and policy objectives. (i.e.,the program).Options for conducting M&V are defined in
2. Select appropriate evaluation approach(es)and prepare a the International Performance Measurement and Verification
program evaluation plan that takes into account the critical Protocol(IPMVP)and include two end-use metering options,
evaluation issues and the expectation for reliability(certainty) billing regression analysis,and computer simulation.This
of evaluated impacts. approach determines gross savings values;net savings can
3. Define data collection requirements. be determined with program-wide adjustments to the gross
savings values.
FIGURE ES.3: Evaluation activities workflow
PLANNING
OBJECTIVES AND
METRICS VERIFICATION EVALUATED SAVINGS
EVALUATION DETERMINE FIRST-YEAR
APPROACHES GROSS AND/OR NET SAVINGS
DATA DETERMINE LIFETIME
COLLECTION SAVINGS
DETERMINE NON-ENERGY
BENEFITS
DETERMINE COST
EFFECTIVENESS
xvi www.seeaction.energy.gov December 2012

b.Deemed savings values:stipulations based on historical 6. Determine the program's cost-effectiveness using one or more
and verified data(in some cases using the results of prior M&V of the common cost-effectiveness tests. Inputs into these tests
studies).Similarly,deemed savings calculations are standardized are the lifecycle net or gross energy and demand savings and
algorithms. Both deemed savings values and deemed savings possibly one or more non-energy benefits.See Appendix B for
calculations should only be used with well-defined energy effi- an overview of cost-effectiveness analyses.
ciency measures that have documented and consistent savings
values.This approach determines gross savings values or net The evaluation approaches described in this guide are often referred
savings values,if net-to-gross ratios are included in the deemed to as"bottom-up"approaches because they add up the savings
savings values or calculations. from measures and projects to determine program impacts,and
C.Large-scale consumption data analysis:uses metered energy they add up the impacts of programs to determine total portfolio
use data to compare the energy use of the program participants impacts.Another evaluation category,called"top-down,"uses
with the energy use of a control group.The control group can be approaches that rely on energy consumption data or per-unit energy
either program nonparticipants,as is the case with randomized consumption indicators(e.g.,energy consumption per-unit of output
controlled trials,or participants,as is the case with some quasi- or per person)defined by market sector, utility service territory,or
experimental methods. If the program participants are used, a geographic region(e.g.,a state or region).Top-down evaluation
their energy use before the program and after the program are is not commonly used for evaluation of efficiency programs and
compared;in effect,this means that each participant is his/ portfolios,although interest in the approach is growing,and it has
her own non random control group.All of these methods can advantages over bottom-up evaluations.A section of Appendix B
provide results that are either gross or net savings values. covers top-down evaluation.
In some cases,the three approaches listed above are combined, E$.5.3 Evaluation Planning Characteristics
particularly the deemed savings and M&V approaches.Portfolios and Frameworks
of programs also often use different approaches for different The following are best practice characteristics for evaluations:
programs to determine total portfolio savings.Multiple-year Evaluation is integral to a typical cyclic planning-implementa-
programs may also conduct detailed measurement-based studies tion-evaluation process.Therefore,evaluation planning is part
(e.g.,M&V)for one year of the program and then apply the of the program planning process,including the alignment of
savings values(deemed savings)for other program years. implementation and evaluation budgets and schedules.This is
3. Convert,as needed,first-year gross program energy(and done so that evaluation efforts can support program imple-
demand)savings to first-year net program savings using a range mentation and provide timely evaluation results for improving
of possible considerations as discussed in Chapters 3 and 5. existing programs and informing future program and energy
4. Determine lifetime savings,which are the expected energy resource planning.See Figure ES.4.
(and demand)savings over the lifetime of the measures that The evaluation process is designed to support the policy goals
are implemented in the efficiency program.These savings are of the energy efficiency programs being evaluated by providing
usually calculated by multiplying the first-year annual energy appropriate documentation of progress toward the goals,
use reduction associated with the subject measures by the as well as feedback required by program administrators and
expected life of these measures with possible consideration implementers to continuously improve the programs and plan
of factors such as performance degradation or in some cases future efforts.
consideration of rebound(an increased level of service that Evaluation budgets and resources are adequate to support,
is accompanied by an increase in energy use as a result of a over the entire evaluation,the evaluation goals and the level
program).Section 7.3 discusses savings persistence. of quality(certainty)expected in the evaluation results.
5. Determine non-energy benefits(NEBs)using a range of Reported values for metrics are those that are"most likely"
subjective and objective analytical tools. Determining avoided and not biased to be overly conservative or overly aggressive.
emissions,which is the primary NEB addressed in this guide, Evaluations use the planning and implementation structure
is discussed in Chapter 6. Evaluating other NEBs is discussed described in this guide,as well as the definitions provided for
in Section 7.9. evaluation terms.
• Energy and demand savings calculations follow one or more
of the approaches defined in this guide.
December 2012 www.seeaction.energy.gov xvii

FIGURE ESA Evaluation is integral to a typical Portfolio cycle EM&V plan.This plan indicates the major evalua-
cyclic planning-implementation-evaluation process tion activities that will be conducted during the evaluation cycle
(typically one,two,or three years). It includes the budget and
allocation among the programs,measures,and market sectors,
as applicable.
14j.4 Evaluation activity-specific detailed plans.Evaluation plans
Z ,� are created for each of the major evaluation activities(typically
k, the evaluation of an energy efficiency program but may include
studies such as market assessments)in a given cycle prior to
0 the time each activity is launched.
• Project-specific plans. Project-specific plans may be required
a N for custom project sites that are analyzed and inspected.
Also complementary to this hierarchy of planning documents is
a reporting structure that can include individual site evaluation
reports,program reports,and annual portfolio reports.
Another typical resource document for large-scale efficiency portfolios
4o1 (such as those for a state or regional consumer-funded efficiency
44(14TE PR0GRAMS program)is a technical reference manual(TRM).A TRM is a database
of standardized,state-or region-specific deemed savings calculations
and associated deemed savings values for well-documented energy
efficiency measures.Energy efficiency program administrators and
• Evaluations are complete,readable,fair,accurate,transparently implementation contractors use TRMs to reduce evaluation costs
documented, relevant,and actionable,as well as balanced
and uncertainty.
between certainty of results and costs to achieve the results. ES.5.4 Evaluation Planning Issues
They also follow the American Evaluation Association's guiding The evaluation requirements described in each of the planning
principles,which are listed in Section 7.6. documents listed above are determined by the program objectives,
With the above characteristics in mind,individual entities can regulatory mandates(if any),expectations for quality(i.e.,reliability)
define their own policy-specific program evaluation requirements.
of the evaluation results,available budgets,timing of reporting dead-
Jurisdictions such as states can establish and document their evalua-
lines,intended uses of the evaluation results,and other factors that
lion requirements in a hierarchy of documents.A useful structure of
can vary across jurisdictions and programs.In this guide(Chapter 8),
planning documents includes the following(see Figure ES.S): 14 key evaluation planning issues are presented and discussed to
help define policy-specific program evaluation requirements:
• Evaluation framework.A framework is a primary docu- 1. What are the policy and/or regulatory goals that are the basis
ment that lays out evaluation principles,metrics,allowable for the efficiency programs,and what are the evaluation
approaches,definitions,and metrics for determination of gross objectives, metrics,and research issues that support the
and/or net savings,reporting requirements,schedules,and program policies and/or regulations?
the roles and responsibilities of various entities.An evaluation 2. What are the evaluation principles that drive the effort?
framework document tends to be"fixed"for several years, 3. What is the scale and budget of the evaluation effort?
but of course can be updated periodically. It often sets the
expectations for the content and scope of the other evaluation 4. Who will conduct the evaluations,how is an independent
documents.This is perhaps the principle document that all evaluation defined,and what are the relative EM&V roles
stakeholders can focus on and provide high-level input to—the between implementers,evaluators, regulators,stakeholders,
and others?
"forest versus the trees"of evaluation planning.
xviii www.seeaction.energy.gov December 2012

5. Is performance determined on the basis of net or gross
savings?What factors are included in defining net savings?
6. What are the baselines against which savings are determined?
7. What is the reporting"boundary"?Are transmission and
distribution(T&D)losses included,and how"granular"will the
results be?
8. What are the schedules for implementing the evaluation
and reporting?
9. What impact evaluation approaches will be used?
10. What are expectations for savings determination certainty
(confidence and precision)?
11. Which cost-effectiveness tests will be used?
12. How are evaluated savings estimates applied—looking
back/going forward?
13. What are the data management strategies?
14. How are disputes addressed?
FIGURE ES.S: Hierarchy of EM&U planning documents
TIMEFRAME COVERAGE
Multiple EM&V Region, State,
Year FRAMEWORK or Program
Administrator
Annual or
PORTFOLIO Region, State,
Multiple Year CYCLE EM&V or Program
PLAN Administrator
As Required EVALUATION Program
(e.g., annual) RESEARCH PLAN or Portfolio
As Required Project
(e.g., annual) or Site
December 2012 www.seeaction.energy.gov xix

Executive Summary: Notes
1"Uniform Methods Project."(2012). U.S.Department of Energy.
wwwl.eere.energy.gov/deployment/ump.html.
2 Although,as discussed in this guide,this sentiment needs to be
tempered with a quote that some attribute to Albert Einstein:
"Not everything that can be counted counts,and not everything
that counts can be counted."
xx www.seeaction.energy.gov December 2012

Chapter 1
Introduction
1.1 GUIDE OBJECTIVE EXPECTATIONS AFTER READING
Jurisdictions and organizations(e.g.,state agencies,regulatory bodies, THIS GUIDE
utilities,efficiency portfolio administrators)can use this guide as
both a primer on efficiency impact evaluation and for defining their After reading this guide,the reader will be able to define
own institution-specific,general evaluation requirements as well the basic objectives,structure,and evaluation approaches
as specific impact evaluation requirements.While each jurisdiction or that can be used to plan and conduct impact evaluations of
entity will need to define its own evaluation requirements,this guide efficiency programs.The reader will also be able to support
provides a structure,a set of evaluation approaches,suggestions on and provide input for an energy efficiency evaluation framework
key evaluation issues,and definitions that can be applied to a variety (general guidance documents)and review impact evaluation
of policy situations. plans and reports.
Applying the information in this guide can be particularly helpful
This guide provides the following:
for jurisdictions and organizations just starting or ramping up their
efficiency and evaluation activities. By using standard approaches • Policy-neutrals descriptions and guidance for planning and
and terminology,developed through 30-plus years of efficiency conducting impact evaluations of end-use efficiency programs6
program evaluation experience,costs for starting up an evaluation to determine energy and demand savings
effort and moving"up the learning curve"can be reduced.Use of • Information on determining energy and demand savings,
common approaches and terminology can also support comparison as well as avoided emissions that result from energy
of efficiency programs in different jurisdictions and facilitate the efficiency programs
implementation of"cross-border"energy efficiency and/or green- . Discussions about issues encountered with planning and
house gas and other air emissions mitigation programs. implementing impact evaluations
1.2 SUBJECTS COVERED IN THIS GUIDE
• A planning process for impact evaluations including a
recommended hierarchy of documents and evaluation reports
This 2012 guide is an update to the 2007 National Action Plan for
• Background on other types of energy efficiency evaluations
Energy Efficiency Model Energy Efficiency Program Impact Evaluation
Guide.3 It includes new and updated material based on feedback • A glossary of evaluation terms
received on the 2007 guide and lessons learned from impact evaluations • A list of other reference documents and resources on energy
conducted during the last five years. efficiency evaluation.
This guide focuses on bottom-up evaluations4 of the impacts—pri- In practical terms,evaluation planners can use this guide to do
marily energy,demand,and emissions savings—of energy efficiency the following:
programs implemented in facilities,and for which energy and demand . Define the questions and hypotheses that the evaluation effort
savings are the primary objectives.Therefore,the guide helps users is intended to address
determine the end-use fossil fuel(e.g.,natural gas)and electricity a Identify appropriate evaluation approaches and methods
savings from programs that encourage lighting,space conditioning, that,from a budgetary perspective,balance the value of the
process approaches,and similar energy efficiency strategies in resi- information provided by impact evaluations with the costs to
dential,commercial,institutional,and industrial facilities.In addition, provide such information at an acceptable level of accuracy
while not a focus of this guide,some guidance is provided in Chapter
7 on documenting non-energy benefits and evaluating market transfor- ' Set realistic expectations among the evaluation process
mation,behavior,training,and behavior-based programs.Appendix stakeholders regarding the nature and practical value of results
B also has sections on market effects and process evaluations,top-down to be delivered,the timing of when evaluation results can be
evaluations,and cost-effectiveness analyses. available,and the expected quality of quantitative estimates of
program impacts
• Set appropriate schedules and budgets that reflect the desired
level of certainty expected in the results.
December 2012 www.seeaction.energy.gov 1-1

•PLANNING ISSUES Members of the energy efficiency community looking for
the following:
While reading this guide's first seven chapters,keep in — Common terminology and definitions
mind the 14"evaluation planning"issues listed in — A central reference that provides guidance and also lists
the Executive Summary and addressed in Chapter 8 with publicly available best practices resources
respect to preparing an evaluation plan. — An understanding of the mechanisms for determining the
potential value of energy efficiency as an emissions avoid-
ance strategy
It is also important to indicate what the guide does not cover: Expert evaluation practitioners looking to provide introductions
• It is not sufficiently detailed to be the only resource for plan- and summaries of evaluation topics to those who do not have
ning or conducting evaluations of specific programs. Rather, their expertise.
the guide provides high-level guidance,identifies issues,and 1.4 SOURCE DOCUMENTS
directs users to resources for defining policy and program-
specific requirements and details. For example,it does not The information in this document is a summary of definitions,
describe specific data collection and analysis options,although approaches,and best practices developed during more than
Appendix C does list documents where this information can be 30 years of energy efficiency program implementation and evaluation.
found for various program types and technologies.? This experience and expertise is documented in numerous guides,
• It is not a guide describing how to perform feasibility studies protocols,papers,and reports.More information on these documents
or potential studies,which are intended to assess potential and other evaluation resources is included in footnoted references
savings and benefits from future energy efficiency projects or throughout the guide and in Appendix C.
programs, respectively. Instead,this guide can be used to help
define and conduct studies that inform on what has been or is
being accomplished with existing programs. USING THIS GUIDE
1.3 GUIDE STRUCTURE AND HOW TO USE Policymakers and those looking for the"basics": Read the
THIS GUIDE Executive Summary and first three chapters and refer to the
Table 1.1 at the end of this chapter provides a summary of the guide appendices for overviews of other evaluation types,defini-
contents and suggestions for which chapters different audiences will tions,and references.
find of interest.
Experienced evaluation planners:Go straight to the evaluation
This guide's intended audience includes the following: considerations chapter(Chapter 7)and the planning chapter
• Program and evaluation managers looking for basic guidance— (Chapter 8)and use the rest of the document as a reference.
or a "road map"—on approaches and key issues including:
Readers new to evaluation or energy efficiency:Read the
— Defining and documenting net and gross energy and entire document.
demand savings
— Documenting avoided emissions
— Comparing demand-and supply-side resources
• Energy system resource planners and demand forecasters
looking for how end-use efficiency impact evaluation strategies
and results can be effectively used in resource planning efforts
• Program designers looking to understand how their programs
will be evaluated and the benefits they can receive from evaluations
• Policymakers and regulators looking for a basic understanding
of evaluation objectives,processes,and issues
1-2 www.seeaction.energy.gov December 2012

TABLE 1.1: Summary of Guide Content and Intended Audience for Each Part of the Guide
INTENDEDPART CHAPTER .
Part 1 Executive Summary Readers interested in a brief The Executive Summary provides an overview of
summary and introduction to impact evaluation,with discussion of the importance
impact evaluation and types of evaluations,the impact evaluation
process,key issues,and evaluation planning.
Part 2 Chapter 1: Introduction Readers who want an overview The Introduction describes the guide's objective,
of evaluation and the key a review of what is and is not covered in the guide,
Chapter 2: Energy aspects of impact evaluation and recommendations for how to use the guide.
Efficiency Program Chapter 2 describes the objectives of evaluation
Evaluation Overview
and provides definitions of different efficiency
Chapter 3: Impact program types and evaluation categories.Chapter 3
Evaluation Basics summarizes the basics,processes,and approaches
associated with impact evaluation.
Part 3 Chapter 4:Calculating Readers who want additional Chapter 4 covers the three categories of approaches
Energy Savings detail on impact evaluation for determining energy and demand savings.Chapter
approaches 5 defines net savings terms and uses,briefly describes
Chapter 5: Determining methods for determining net savings,and discusses
Net Energy Savings issues associated with the use and calculation of
Chapter 6:Calculating net savings.Chapter 6 provides approaches for
Avoided Air Emissions determining avoided air emissions associated with
efficiency programs.
Part 4 Chapter 7: Impact Program implementers, Chapter 7 provides background on topics associated
Evaluation evaluators,and managers/ with implementing impact evaluations that are
Considerations regulators of evaluations not covered in other chapters,such as persistence,
Chapter 8: Impact looking for guidance on key demand savings,controlling uncertainty, non-energy
evaluation issues and planning benefits,program costs,and evaluation issues
Evaluation Planning of evaluations as well as readers associated with certain unique program types.
with a background in evaluation
may want to go directly to these Chapter 8"brings it all together"and describes how
chapters the basics and details described in earlier chapters
can be used to plan evaluation efforts.
Part 5 Appendix A:Glossary Readers interested in standard The appendices provide resources and further
energy efficiency evaluation background on evaluation topics.
Appendix B:Other definitions and reference
Evaluation Categories materials used in the evaluation
and Approaches
industry as well as summaries
Appendix C: Resources of process,market evaluations,
References cost-effectiveness analyses,and
top-down evaluation
December 2012 www.seeaction.energy.gov 1-3

Chapter 1 : Notes
3 National Action Plan for Energy Efficiency.(2007).Model Energy
Efficiency Program Impact Evaluation Guide. Prepared by Steven R.
Schiller,Schiller Consulting, Inc.www.epa.gov/eeactionplan.
4"Bottom-up"evaluation involves adding up the savings from
measures,projects,and programs to estimate total portfolio
impacts.Another evaluation category,termed"top-down,"refers to
approaches that rely on energy consumption data or per-unit energy
consumption indicators(e.g.,energy consumption per-unit of output
or per person)defined for a market sector,utility service territory,
or jurisdiction as the starting point for savings determination using
macro-economic approaches.
5 Because the guide is a policy-neutral document,evaluation plans
must address any jurisdiction-specific policy requirements.
6 The guide does not cover transportation-related energy efficiency
programs.
7 In addition to guidance documents,the planning and implementa-
tion of impact evaluation activities requires skilled and experienced
practitioners.
1-4 www.seeaction.energy.gov December 2012

Chapter 2
Energy Efficiency Program Evaluation Overview
Chapter 2 provides an overview of the three main categories of energy efficiency program evaluation.The chapter also
makes the distinction between evaluations for individual energy efficiency projects and multifaceted efficiency programs.
Because this guide focuses on end-use energy efficiency program evaluation, some background on different program
categories is also provided.The last sections cover the importance and objectives of energy efficiency evaluation.
2.1 ENERGY EFFICIENCY PROGRAM 2.1.2 Efficiency Program Categories
CATEGORIES AND DEFINITIONS There are many types of energy efficiency programs and several
Before describing the different categories of evaluation and its approaches to differentiating them.One approach is to divide
importance and objectives,this section provides some context programs into two categories:voluntary and mandatory.Mandatory
regarding categories of efficiency programs that are the subject programs involve codes and standards that require mandated levels
of evaluations,as well as definitions of the savings hierarchy. of efficiency in buildings and/or products(e.g.,equipment or appli-
ances).Voluntary programs involve a wide range of mechanisms to
2.1.1 Policy Context of Energy Efficiency Programs incent consumers to use energy more efficiently.
The evaluation of energy efficiency programs should support the
policy goals of the programs.Thus,understanding policy goals,and Voluntary programs can be defined as including the following
the context in which the programs are being implemented,affects subcategories and objectives:
program evaluation. Policy goals can vary widely;however,a major- Resource acquisition.The primary objective of this program
ity of states now have policies in place that establish specific energy category is to directly achieve energy and/or demand savings,
savings targets for energy efficiency programs provided to customers and possibly avoid emissions,through specific actions.This
by their utilities or related organizations.$The following are three category includes activities such as rebate and direct-install
common ways in which states have set their efficiency goals—by programs for energy-efficient equipment,specific operational
legislation,regulation,or voter initiative. or maintenance actions(e.g.,boiler tune-ups, building
• All cost-effective energy efficiency.This requires that the state commissioning),and behavior-based programs that encourage
or its utilities must acquire all energy efficiency measures that consumers to adopt energy and demand savings practices.
are less expensive than energy supply options. Funding for These later programs typically include outreach,education,
efficiency programs can be drawn from dedicated utility bill rewards, benchmarking,and/or feedback elements.
surcharges and/or the budgets that utilities would otherwise Market transformation(MT).The primary objective of this
use to procure a more expensive energy supply. program category is to change the way in which energy
• Energy efficiency resource standard(EERS).An EERS requires efficiency markets operate(e.g.,how manufacturers,distribu-
that a percentage of the resources used by utilities to supply tors, retailers,consumers,and others sell and buy energy-
their customers must come from energy efficiency.An EERS related products and services),which tends to result in more
policy can mandate that the utility increase the percentage indirect energy and demand savings. Education and training
of energy efficiency incrementally over a number of years (E&T)programs and programs that support the development
(e.g., 1%per year)or achieve a specific target percentage of or compliance with codes and standards(C&S)are examples
by a future date(e.g.,20%by 2025): of market transformation activities.These programs indirectly
• Target spending budget.This requires that an efficiency result in energy savings.To a large extent,all programs can
program administrator spend a certain amount of money on be considered market transformation programs in that they
energy efficiency portfolios and maximize energy or peak sav-
involve a change in how energy efficiency activities take place
ings within these portfolio budgets. in the marketplace.
• Multiple objectives.Programs can include some or all of the
above-listed objectives.
December 2012 www.seeaction.energy.gov 2-1

APPLICATIONSSOME 2.1.3 Savings Hierarchy
EFFICIENCY EVALUATIONS The starting point for evaluating energy and demand savings,at least
with bottom-up evaluation approaches,is a savings hierarchy for
• Utility-administered energy efficiency programs energy efficiency actions,as shown in Figure 2.1.This figure shows
• Government efficiency programs,either for public the energy efficiency actions in the following order:
facilities or for private-sector incentive programs Energy efficiency measure:at an end-use energy consumer
• Independent system operator(ISO) programs to reduce facility,an installed piece of equipment or system;a strategy
demand(e.g.,a forward capacity market). intended to affect consumer energy use behaviors;or modification
• Air pollution and greenhouse gas mitigation programs of equipment,systems,or operations that reduces the amount
that rely on efficiency actions of energy that would otherwise have been used to deliver an
equivalent or improved level of end-use service. Examples
• Private company programs include lighting retrofits, HVAC retrofits,and commissioning.
• Energy service company contracts Project:an activity or course of action involving one or multiple
energy efficiency measures at a single facility or site. Examples
include home retrofits and commercial new construction projects.
This guide focuses on documenting the impacts of resource acquisi- Program:an activity,strategy,or course of action undertaken
tion programs,including directly achieved energy and demand by a program implementer or administrator. Each program
savings and related emission reductions.Section 7.10 of this guide is defined by a unique combination of program strategy,
briefly discusses evaluation of market transformation programs, market segment,marketing approach,and energy efficiency
including education and training and codes and standards programs. measure(s). Programs consist of a group of projects with similar
It should be noted that while a program may have just one primary characteristics and installed in similar applications. Examples
objective,there are often secondary objectives that are integral include a utility program to install energy-efficient lighting in
to the program's overall success.This is frequently the case when commercial buildings,a developer's program to build a
resource acquisition and market transformation objectives are subdivision of homes that exceed common practice,or a state's
involved.With respect to impact evaluation,it is more important to effort to improve compliance with energy efficiency codes.
focus on the performance goals to be assessed and establish metrics
than to categorize individual program types.
End-use(consumer)energy efficiency is part of the very general FIGURE 2.1: Hierarchy of energy efficiency activities
category of activities known as demand-side management(DSM).
Demand-side management programs are designed to encourage
consumers to modify their level and pattern of energy use.Another
category of DSM is demand response(DR),defined by the Federal
Energy Regulatory Commission(FERC)as"a reduction in the consumption
of electric energy by customers from their expected consumption in
response to an increase in the price of electric energy or to incentive PROGRAMS
payments designed to induce lower consumption of electric energy."9
Demand response programs employ energy rate design(pricing),
customer incentives,and technology to enable customers to change
their demand in response to system conditions or prices. PROJECTS
While this guide does not specifically address DR programs,the basic
evaluation approaches and planning process explained here can be
applied to DR with the understanding that the emphasis for DR program MEASURES
evaluation is demand savings.Demand savings definitions and evaluation
techniques are highlighted in Sections 7.2 and 7.10.5.
2-2 www.seeaction.energy.gov December 2012

• Portfolio:either(1)a collection of similar programs addressing WHY CONDUCT EVALUATIONS?
the same market(e.g.,a portfolio of residential programs),
technology(e.g.,motor efficiency programs),or mechanisms The reasons to do an evaluation can be summarized in two
(e.g.,loan programs),or(2)the set of all programs adminis- words:improvement and accountability. Evaluations provide
tered by one organization,such as a utility. information that can help improve programs and demonstrate
2.2 PROGRAM EVALUATION CATEGORIES internal and external accountability for the use of resources.
AND IMPACT EVALUATION DEFINITIONS Program evaluations provide timely information to improve
The variety of evaluation activities that are associated with energy not only program implementation,but also the design of
efficiency can be categorized in several different ways,one of which future programs and individual energy efficiency projects.
is to define evaluations as either formative or outcome. Formative They can answer the following questions:
evaluations are associated with helping efficiency programs be as • Are the program and the projects that make up the
effective as possible.Outcome evaluations are associated with program achieving their goals?If so,how and why?
documenting program results. However,the most common way . How well has the program/project worked?
to categorize efficiency evaluations is as impact,process,or market
• What changes are needed to improve the
evaluations.These are defined as follows(with the first two
program/project?
described in more detail in Appendix B).
• Impact evaluations:outcome evaluations of the changes • What is the program's impact on actual projects
attributable to an energy efficiency program.While impact and future projects?
evaluations usually focus on determining the quantity of • Should the program/project be replicated,adjusted,
changes in energy use and demand associated with a program, or cancelled?
the calculation of non-energy benefits(or co-benefits)such
Program evaluations also provide an understanding of
as avoided emissions and job creation that directly or indirectly
result from a program can also be an output of impact evaluations. the following:
Impact evaluations often support cost-effectiveness analyses • Program approaches that are most and least effective,
that document the relationship between the value of program as well as how to improve future programs
results(i.e.,energy,demand,and emission savings)and the • Where to focus for greater savings
costs incurred to achieve those benefits.Cost-effectiveness • Actual values that can be used in future estimates of
(sometimes called cost-benefit)analyses may also take into benefits(e.g.,estimates of energy savings per square
account market evaluation results considering a program's foot of office space).
short-and long-term market effects.
• Process evaluations:systematic assessments of an energy
efficiency program.Their purpose is to document program Market evaluations:a very broad category of activities that
operations and identify and recommend improvements to document aspects of the marketplace with respect to energy
increase the program's efficiency or effectiveness for acquiring efficiency.One particular type is a market effects evaluation,
energy resources while maintaining high levels of participant which characterizes changes in the structure or functioning of
satisfaction. For example,process evaluations can include an a market or the behavior of market participants that resulted
assessment of program delivery,from design to implementation, from one or more program efforts. Market effects evaluations
to identify bottlenecks,successes,failures,constraints,and can include projections of impacts that a market could have on
potential improvements.Timeliness in identifying opportunities future energy efficiency efforts. If the evaluation's goal is to
for improvement is essential to making corrections along the assess cost-effectiveness for stakeholders or regulators,excluding
way.Process evaluations also provide a backdrop for interpreting the measurement of market effects could result in underesti-
the results of impact evaluations. mating(or possibly overestimating)a program's overall benefits
or cost-effectiveness.
December 2012 www.seeaction.energy.gov 2-3

TABLE 2.1: Summary of Evaluation Categories and Types
EVALUATIONEVALUATION PHASE AT WHICH IT IS . ASSESSMENT
-
. -
Market Assessment Analyses
Market, Portfolio, Program
(includes characterization, baseline)
Pre-program Planning Phase
Potential or Feasibility Analyses
Portfolio, Program, Project
Formative Portfolio, Program, Project
Implementation Phase Process Evaluations Portfolio, Program
and Ongoing and/or After
Program Implementation Market Effects Assessments Portfolio, Program
Impact Evaluations Program, Project, Measure
Implementation Phase—
Outcomes Ongoing and/or After Market Effects Evaluations Market, Portfolio
Program Implementation
Cost-Effectiveness Analyses Portfolio, Program,Project
The following are example questions that could be used to implicitly bundled with impact evaluation.Table 2.1 summarizes
determine market effects: these categories of efficiency evaluation,although not all of these
— Did a program encourage more vendors to offer energy- different evaluations are necessary for every program or portfolio.
efficient products,and will there thus be future efficiency
2.2.1 Evaluation Definition
benefits associated with such increased availability of
products? Evaluation is the conduct of any of a wide range of assessment
studies and other activities aimed at determining the effects of a
— Did a voluntary incentive program prove that a new tech- program(or a portfolio of programs).This includes understanding
nology is viable,cost effective,and accepted by consumers, or documenting program performance,program or program-related
and therefore make it possible for this technology to be markets and market operations,program-induced changes in energy
included in a future building code or appliance standard? efficiency markets,levels of demand or energy savings,or program
There are other types of market evaluations:market assessment
cost-effectiveness.While this guide focuses on evaluations of
studies used to determine current practices for the purposes individual programs,the basic concepts can be applied to portfolios.
of establishing measure,project,or program baselines;and Measurement and verification(M&V)is another term often used
potential studies used to estimate the technical,economic,or when discussing analyses of energy efficiency activities. M&V can be
market-based potential of increasing the amount of energy a stand-alone activity or it can be a subset of program impact evaluation.
efficiency for various products and services. In either case,it is associated with the documentation of energy
While this document focuses on impact evaluation,all types of (and/or demand)savings at individual sites or projects using one
formative and outcome evaluations are not mutually exclusive,and or more options that can involve measurements,engineering calcula-
there are benefits to undertaking more than one type at a time and tions,statistical analyses,and/or computer simulation modeling.
integrating the data collection and analyses functions.Thus, process These options are defined in the International Performance
evaluation and market effects evaluation often end up explicitly or Measurement and Verification Protocol(IPMVP).10 Generally speaking,
2-4 www.seeaction.energy.gov December 2012

the differentiation between evaluation and project M&V is that of contractors and administrators responsible for implementing
evaluation is associated with programs(or portfolios)and M&V is efficiency programs.
associated with projects.Contractors and the owners of facilities tend 2. Help understand why program-induced effects occurred
to be interested in only M&V on their own project(s),while program and identify ways to improve current programs and future
administrators are interested in evaluation of their programs and programs.The role of evaluation can go well beyond simply
portfolios. documenting savings to actually improving current and future
As discussed in later chapters, M&V is also one of the three programs.If applied concurrently with program implementation,
approaches used for program evaluation,where M&V techniques evaluations can provide information in real time to allow for
as-needed course corrections. Evaluation fosters more-effective
are typically used to determine the savings from a sample of projects
programs and can justify increased levels of energy efficiency
(versus a census),with the results applied to the entire program
investment as along-term, reliable energy resource. Perhaps
population of projects.The other two evaluation approaches are
the imperative for conducting evaluation is best described by
deemed savings(which does not involve any project-specific mea-
a quote attributed to John Kenneth Galbraith:"Things that are
surement)and large-scale consumption data analysis. Both are
measured tend to improve:'
typically applied to all of the projects(or sites)in a program.
3. Support energy demand forecasting and resource planning
The term evaluation,measurement,and verification(EM&V)is by understanding the historical and future effects of energy
frequently seen in evaluation literature and is a catchall term for efficiency as compared to other energy supply and demand-
determining both program and project impacts. side resources.As efficiency has become a more important
energy resource in state and regional energy plans,an objective
2.3 OBJECTIVES AND IMPORTANCE of impact evaluations can be to support state and regional
OF EVALUATION energy forecasting and resource-planning efforts.Understanding
The first step in the evaluation process is to define the objectives and supporting the needs of forecasters and planners(and
for given evaluation activities.This leads to defining the metrics, their data formats and definitions)can thus be an important
assumptions,approaches,budgets,and other characteristics of the consideration when defining end-use efficiency program
evaluation effort. Evaluations have three overarching objectives: evaluation metrics and reporting requirements. In addition,
Document the benefits/impacts of a program and determine evaluation can support resource planning through projections
1.
whether the program(or portfolio of programs)met its goals. of non-energy benefits,specifically emissions profiles for
Rigorous evaluations help ensure that programs are cost-effective planning how to meet air quality and greenhouse gas
and that benefits(e.g.,energy savings,avoided emissions) mitigation objectives(see Chapter 6).
are both "real"and sustained over time.This often includes There are several technical and policy barriers to the full use
assessment of compliance with regulatory requirements of cost-effective energy efficiency,and to the incorporation of
associated with programs funded by the public(or energy efficiency programs into energy resource portfolios.One of these
consumers). Energy efficiency impact evaluations are oriented barriers is proving that energy efficiency"can be counted on"or is
toward developing retrospective estimates of energy savings "real."Consistent,complete,accurate,and transparent evaluation
attributable to a program,in a manner that is defensible mechanisms for documenting energy and demand savings,as well as
in regulatory proceedings conducted to ensure that public non-energy benefits such as avoided emissions,address this barrier.
funds are properly and effectively spent. Regulators support Indeed,having effective evaluation policies, processes,and trained
evaluation activities because of their interest in documenting personnel in place to document the energy and non-energy benefits
total savings,assessing the cost-effectiveness of efficiency of energy efficiency programs is critical to the success of energy
compared to generation alternatives,and assessing savings efficiency,emission,and climate change-mitigation programs that
attribution(e.g.,the contributions of efficiency portfolio must prove their value and worthiness for continued investment.
administrators in achieving savings versus the influences of
common practice,end-user self-motivation,or codes and Evaluation is thus not a goal unto itself;it should be viewed as one
standards).With respect to this last objective,evaluation can part of a continuous,and usually cyclic,process of program planning,
also be used explicitly for retrospectively determining the implementation,and evaluation.The results of impact evaluation
performance(and resulting payments, incentives,or penalties) studies do not stand alone but are used as inputs into planning
December 2012 www.seeaction.energy.gov 2-5

and improving future programs.As shown in Figure 2.2,there is a
continuum of strategies associated with moving energy efficiency
actions from the research,development,and demonstration(RD&D)
stage to an increasing level of adoption,and to ultimately have
them become standard practice or be mandated through codes
and standards.With public policy orientated toward accelerating
the success and fulfillment of these strategies and greater energy
savings,evaluation is a tool that supports the acceleration through
documentation,feedback,and energy resource planning.
FIGURE 2.2: Continuum of energy efficiency actions
DEPLOYMENT TRANFORMED MARKETS
Voluntary implementation . Standard practice
of projects and programs: or
outreach, education, • Codes and standards
and subsidies.
RD&D
Research For example:
Development • Incenting customers, distributors,
Demonstration and manufacturers
• Mass market and individual
market strategies
2-6 www.seeaction.energy.gov December 2012

Chapter 2 : Notes
8 Nowak,S.; Kushler, M.;Sciortino, M.;York,D.;Witte,P. (June 2011).
Energy Efficiency Resource Standards:State and Utility Strategies
for Higher Energy Savings.American Council for an Energy-Efficient
Economy(ACEEE). Report Number U113.www.aceee.org/
resea rch-report/u 113.
9 Federal Energy Regulatory Commission(FERC).(June 2010).
National Action Plan on Demand Response. Docket No.AD09-10.
Prepared by FERC staff.www.ferc.gov/legal/staff-reports/
06-17-10-demand-response.pdf.
10 Efficiency Valuation Organization(EVO). International Performance
Measurement and Verification Protocol(IPMVP).(multiple dates).
www.evo-world.org.The IPMVP is an international M&V guidance
document. It is discussed in Chapter 4 and references are
in Appendix C.
December 2012 www.seeaction.energy.gov 2-7

Chapter 3
Impact Evaluation Basics
Chapter 3 is a stand-alone introduction to the basic concepts associated with energy efficiency impact evaluation.
It introduces the evaluation process,the role of evaluators versus administrators of efficiency programs, and some
key metrics. Also presented is the concept of savings determination based on a counterfactual situation and the
fundamental resulting issue of evaluation being the balancing of evaluation costs with the reliability of savings
estimates.The chapter concludes with a brief introduction to the approaches used for determining energy and
demand savings: deemed savings, measurement and verification, and large-scale consumption data analysis.
3.1 IMPACT EVALUATION PROCESS EVALUATION PLANNING ISSUES
Impact evaluations determine program-specific induced effects,
which include reductions in energy use(such as kilowatt-hours[kWh] Chapter 8 discusses the evaluation planning process.
and therms)and demand(kilowatts[kW]),and non-energy benefits The planning process is used to decide which(or which
such as avoided air emissions.The basic steps in the evaluation combination)of the evaluation metrics and approaches
process are as follows: defined in this guide should be used.
• Set the program evaluation objectives in the context of the
program policy objectives.
• Select an impact evaluation savings determination approach, Verification would involve confirming that the replaced lamps are
define baseline scenarios,and prepare a plan that takes into 100 watts,that the new CFLs are 23 watts,and that the CFLs were
account the critical issues. installed and are working.As a result of this verification,it can be
• Determine energy and demand savings. confirmed that the project has the potential to save energy,with the
amount of energy saved dependent on how many hours the lamps
• Determine non energy benefits(as needed). operate. Determining how many hours the lamps operate,in this
• Report the evaluation results and,as appropriate,work with case,would be the"documentation"activity(or what some might
program administrators to implement recommendations for call the"measurement,"although as discussed in Section 3.3,savings
current or future program improvements and/or resource cannot be literally measured).
planners and demand forecasters to support their efforts.
In this guide,verification is defined formally as an independent
The program evaluation process begins with defining and assessing assessment that a program has been implemented per the program
the evaluation objectives.Well-defined objectives indicate what design.For example,the objectives of measure installation verification
data need to be collected or developed during the evaluation effort are to confirm(1)the installation rate(number of units installed),
and the scope and scale of effort required for meeting the objectives (2)that the installation meets reasonable quality standards,and(3)
(e.g.,the cost of obtaining the desired information,schedules,labor that the measures meet the program eligibility requirements and
requirements).A key to successful evaluation is the comparison of are operating correctly with the potential to generate the predicted
the costs of evaluation with the value of the information that will come savings.For some programs,it may be that verifying the potential to
from the evaluation, possibly through an iterative planning process generate savings is all that is needed to meet the evaluation objectives,
that balances cost and value. while in many other situations,both verification and documentation
3.1.1 Verification and Documentation of the actual savings value will be required.
Within the impact evaluation process there tends to be two types of Verification may include one-time or multiple activities over the
major activities:verification and documentation.These may be more estimated life of the measures and can overlap with commissioning
aptly called"verifying the potential to generate savings"and"documenting (Cx)or retro-commissioning(RCx)activities,which have similar
(determining)the actual savings! To illustrate the difference between objectives—to ensure the installed equipment is working correctly
the two,consider a project involving replacement of 100-watt and per design. However,verification may not go"as far"as Cx or
incandescent lamps with 23-watt compact fluorescent lamps(CFLs). RCx to ensure the operation of the energy efficiency measure(s).
3-1 www.seeaction.energy.gov December 2012

BASIC IMPACT EVALUATION CONCEPTS activities are completed.These are typically estimates of savings
prepared for program and/or portfolio planning purposes.
• Impact evaluations are used for determining achieved • Claimed savings:values reported by an implementer or
program effects. administrator after the subject energy efficiency activities
• Savings cannot be directly measured,only indirectly have been completed.
determined by comparing energy use and demand • Evaluated savings:savings estimates reported by an independent
after a program is implemented to what they would third-party evaluator after the subject energy efficiency activities
have been had the program not been implemented and an impact evaluation have been completed.These can
(i.e.,the baseline). differ from claimed savings in that an independent third-party
• Successful evaluations harmonize the costs incurred evaluator,to an agreed-to level of rigor,has conducted evaluation
with the value of the information received;in other and/or verification activities.
words,they appropriately balance risk management,
uncertainty,and cost considerations. The implementers and/or administrators usually prepare their
projected savings estimates and claimed savings estimates.They will
thus conduct their own evaluation activities,using their own evalua-
tion staff or consultants,for purposes such as confirming any incen-
3.1.2 The Evaluator's Role and Reporting of Savings tive payments to program participants or contractors and preparing
There are several entities involved in the actual implementation of documentation for internal and external reporting.11 If an indepen-
efficiency projects and programs,and they may each conduct their dent third-party evaluator is used,that evaluator will then conduct
own impact evaluation activities.These entities include the end-use some level of evaluation(verification only or verification and their
energy consumers who have projects installed in their facilities, own data collection/analysis to determine savings)for preparation of
designers,contractors,and program implementers and administra- its own evaluation reports and a realization rate12 comparing evalu-
tors.Some of these entities may only be interested in verification ated savings with projected savings estimates and/or claimed savings
activities or just their own individual projects'measurement and estimates.Of course,the evaluator can and should be brought into
verification,while others will conduct complete program evaluations. the process before any of this work is conducted to participate in
The following definitions help to explain these roles. defining the roles and responsibilities of the administrators,imple-
Entities: menters,and evaluators as well as reporting requirements.The
• Administrator:an entity selected by a regulatory or other designation of"independent"and"third-party"is determined by
government organization to contract for and administer an those entities involved in the use of the evaluations and may include
energy efficiency portfolio within a specific geographic region
and/or market.Typical administrators are investor-owned or
public utilities, nonprofits,and state government agencies. QUALITY ASSURANCE
An administrator could also be a private entity that hires a
company,such as an energy services company(ESCO),to The impact evaluation approaches described in this guide are
implement its efficiency program(s). based on new and unique analyses of energy and demand
savings. However,often there is documentation on energy
• Implementer:an entity selected and contracted with,or and demand savings from analyses prepared independently
qualified by,a program administrator to provide products of the subject impact evaluation.Even though such documentation
and/or services to consumers,either directly or indirectly. was not necessarily prepared per predetermined evaluation
• Independent third-party evaluator:an entity that conducts requirements,it may be sufficient for meeting the evaluation
evaluations and is designated to be independent of the objectives. Using existing documentation in combination with
implementer and administrator. quality assurance guidelines(QAG)can significantly reduce
overall program/evaluation costs. Essentially,a QAG can help
Reported Savings: determine whether indicated savings,and the assumptions
• Projected savings:values reported by an implementer or and rigor used to prepare the existing documentation,can be
administrator prior to the time the subject energy efficiency used in place of new evaluation efforts.
December 2012 www.seeaction.energy.gov 3-2

evaluators retained,for example,by the administrator or a regulator.
Defining the relative roles of the administrator,implementer,and
independent third-party evaluator is another important activity of Savings,or more accurately stated"savings estimates,"
the planning process.Section 8.3.2 of this guide discusses evaluator from energy efficiency measures, projects,programs,and
roles and selection. portfolios are reported at various times in the lifecycle of
3.2 ENERGY AND NON-ENERGY BENEFIT the efficiency activity and with varying degrees of cer-
EVALUATION METRICS tainty.Savings are most commonly reported at two major
milestones—prior to and after the implementation of the
For energy and demand savings(and conceptually for non-energy activity.Savings can also be indicated as first-year,annual,
benefits)the primary metrics are known as gross energy savings and/or lifetime energy or demand savings values.They also
and net energy savings.In this guide,based on industry standard can be indicated as gross savings and/or net savings values.
practice,gross and net energy(and demand)savings are defined Different jurisdictions currently have different names for
as follows: savings reports,what they contain,and whether and what
• Gross energy savings:the change in energy consumption and/ adjustments or evaluation activities take place between
or demand that results directly from program-related actions preimplementation and postimplementation. Ideally,the
taken by participants in an efficiency program, regardless of terms and methods in this guide should be applied. However,
why they participated.This is the physical change in energy use whenever savings are reported,it is critical that the basis for
after taking into account factors not caused by the efficiency the values indicated be made clear.
actions(e.g.,changes in weather or building occupancy).
• Net energy savings:the change in energy consumption
and/or demand that is attributable to a particular energy When energy or demand savings are reported,they are typically
efficiency program. Estimating net energy savings typically estimated for the first year of a program,for a specific number of
involves assessing free ridership and spillover,although this years(e.g.,5,10,15),or for the life of the program's measures.
guide discusses additional considerations. In the efficiency Measure life is the length of time that an energy efficiency measure
industry,free ridership refers to the portion of energy savings is expected to be functional and generating savings. It is a function
that participants would have achieved in the absence of the of equipment life and measure persistence.Equipment life is the
program through their own initiatives and expenditures number of years that a measure is installed and will operate until
(i.e.,the participant would have undertaken the energy-saving failure.Measure persistence refers to the duration of an energy-
activity anyway).Spillover refers to the program-induced consuming measure,taking into account business turnover,early
adoption of measures by nonparticipants and participants retirement of installed equipment,and other reasons measures
who did not claim financial or technical assistance for addi- might be removed or discontinued.Measure life is sometimes
tional installations of measures supported by the program. For referred to as expected useful life(EUL).
instance,a participant undertakes additional energy efficiency
measures due to positive experience with the program,or a The other two main metric categories are non-energy benefits
nonparticipant undertakes such measures based on observing and cost-effectiveness.
a program participant's results. Net savings estimates also • Non-energy benefits(NEBs):the identifiable—although
sometimes include consideration of market effects. sometimes unquantified—non-energy impacts associated with
program implementation or participation;also referred to as
The difference between these two metrics is associated with(1) non-energy impacts(NEI)or co-benefits. Examples of NEBs
attribution of the savings—in other words,the determination of include environmental benefits,productivity improvements,
whether the savings were caused by the program being studied jobs created,reduced program administrator debt and disconnects,
(entirely or partially)or by other influences such as prior year and higher comfort and convenience levels of participants.The
programs or other programs/influences operating at the same time value of NEBs is most often positive,but may also be negative
as the program;and(2)differences in how different entities(e.g.,
(e.g.,the cost of additional maintenance associated with a
regulatory bodies)define net and gross savings.Approaches for sophisticated energy-efficient control system),which is why
determining gross and net savings are summarized in this chapter, some practitioners prefer the term NEls. Potential benefits of
with additional information provided in Chapters 4 and 5. efficiency to the energy system(e.g.,price stability,grid reliability,
3-3 www.seeaction.energy.gov December 2012

and power quality)are"energy-related"but are also often put FIGURE 3.1: True program savings: the counterfactual
into this general category of NEBs.The primary NEB addressed for a residential household energy efficiency program
in this guide is avoided air emissions(see Chapter 6).Section
7.9 has more information on calculating non-energy benefits.
• Cost-effectiveness:an indicator of the relative performance TRUE
or economic attractiveness of any energy efficiency investment PROGRAM
or practice relative to energy supply resources. It is another SAVINGS
metric that is commonly used when reporting the results of
impact evaluations.In the energy efficiency field,the present
value of the estimated benefits produced by an energy efficiency
program is compared with the estimated total costs of the
program in order to determine whether the proposed investment
or measure is desirable from a variety of perspectives(e.g.,
whether the estimated benefits exceed the estimated costs from
a societal perspective or from a program participant perspective).
3.3 FUNDAMENTAL ENERGY EFFICIENCY
IMPACT EVALUATION CONCEPTS: THE
COUNTERFACTUAL AND MANAGING Households Counterfactual:
UNCERTAINTY in Program Exact Same Households,
3.3.1 The Counterfactual Same Time Period, No Program
In theory,the true energy savings from an energy efficiency program
is the difference between the amount of energy that participants in
the program use relative to the amount of energy those same partici- (e.g.,through measurement),but to determine the savings(the blue
pants would have used had they not been in the program(during the shaded area),the energy use that would have occurred without the
same time period).This baseline is called the counterfactual scenario project(the green line)has to be estimated in order to determine a
(see Figure 3.1). However,in practice,we can never observe how value for energy savings.
much energy those participants would have used had they not been
in the program,because at any given time a participant must either As discussed in Chapter 2,an objective of program evaluation is
be in the program or not.Thus,there is no direct way of measuring to produce energy and demand savings values(and,as desired,
energy(demand)savings,because(1)it is not possible to measure a associated non-energy benefits). However,as noted above,these
participant's energy use with and without the program,at the same values are always going to be estimates;the use of these estimates
time;and(2)one cannot measure the absence of energy use. as a basis for decision making can be called into question if their
sources and level of accuracy are not analyzed and described.
Defining this counterfactual scenario represents the fundamental Therefore,evaluation results,like any estimate,should be reported
concept and the greatest challenge to documenting the benefits as"expected values"with an associated level of uncertainty.Most
of energy efficiency.This challenge is met with impact evaluations of the remainder of this guide describes the approaches,issues,and
measuring energy consumption—but,the savings themselves will planning processes that should be considered when addressing the
always be estimates.The savings estimate is the difference between counterfactual challenge and undertaking impact evaluations.
(1)actual energy consumption after a project or program is imple-
mented,and(2)what energy consumption would have occurred 3.3.2 Baselines
during the same period,by the same participants,had the efficiency The baseline is the counterfactual scenario,determined on the basis
project/program not been implemented. of a number of considerations:the evaluation approach being used,
the type of project being implemented,site-specific issues,and
The graph in Figure 3.2 summarizes this estimation process.The blue broader policy-orientated considerations.These considerations usu-
line represents energy use of a building before,during,and after an ally result in one of three different types of baselines being selected
efficiency project is implemented.This energy use can be known for the impact evaluation:existing conditions,common practice,or
December 2012 www.seeaction.energy.gov 3-4

FIGURE 3.2: Energy consumption before, during, and after a project is implemented
BEFORE PROJECT INSTALLED � AFTER PROJECT INSTALLED
•
Estimated Energy Use Without
Efficiency Project
l
W
0
cc
W 4� Energy Savings
Z
W
Energy Use Before
Efficiency Project
Energy Use After
Efficiency Project
TIME
codes/standards. Baselines are discussed throughout this guide with Control group approaches.These are the large-scale consumption
respect to the evaluation approaches and planning for evaluation data analysis approaches that are also described in Chapter 4.
activities.The first section of Chapter 7 provides more information With these approaches,a comparison group's energy use is
on selecting baselines. compared with the energy use of program participants.These
approaches,in most cases,generate estimates of net savings,
3.4 DETERMINING ENERGY AND taking into consideration free ridership and participant spillover,
DEMAND SAVINGS but do not take into account nonparticipant spillover and long-
The three impact evaluation approaches used to determine energy term market effects,which some jurisdictions include in the net
and demand savings can be grouped into two conceptual frameworks: savings determination.
noncontrol group approaches and control group approaches.
3.4.1 Noncontrol Group Impact Evaluation Approaches
• Noncontrol group approaches.These are the deemed savings The following are brief summaries of the two noncontrol group
and M&V approaches defined below and in greater detail approaches:measurement and verification and deemed savings.
in Chapter 4.With these approaches,pre-project(or pre-
program)baseline energy use is defined using one or more Measurement and verification(M&V).Measurement and
of a variety of different methods.This baseline is compared verification is the process of using measurements to reliably
with post-project(or post-program)energy use measurements determine energy and/or demand savings created within an
or assumptions to estimate savings.These noncontrol group individual facility.The International Performance Measurement
approaches generate estimates of gross savings,which require and Verification Protocol(IPMVP),an international M&V
adjustments to determine net savings. guidance document,defines four M&V options used in the
3-5 www.seeaction.energy.gov December 2012

efficiency industry:two end-use metering approaches,energy installed correctly,number of point-of-sale CFLs that were sold).
use data(billing data)regression analysis,and calibrated This approach is only valid for projects with fixed operating
computer simulation. conditions and well-known,documented stipulation values.This
• Deemed savings.Deemed savings are based on stipulated approach involves multiplying the number of installed measures
values,which come from historical savings values of typical proj- by the estimated(or deemed)savings per measure.
ects.A typical source of such historical values are prior year M&V
or large-scale consumption data analysis studies.Unlike the A variant of deemed savings is the deemed savings calculation,
M&V approach,with the use of deemed savings,there are no(or which is one or more agreed-to(stipulated)engineering algorithm(s)
very limited)measurement activities;instead,only the number used to calculate the energy and/or demand savings associated
of measures implemented is verified(e.g.,number of motors with an installed energy efficiency measure.These calculations may
include stipulated assumptions for one or more parameters in the
algorithm,but typically they require users to input data associated
MEASUREMENT AND VERIFICATION with the actual installed measure into the algorithm(s).
VERSUS DEEMED SAVINGS 3.4.2 Control Group Approaches: Large-Scale
For simple,well-defined,efficiency measures whose Consumption Data Analysis (Randomized Controlled
performance characteristics and use conditions are well Trials and Quasi-Experimental Methods)
known and consistent,a deemed savings approach may be
A reliable(precise,unbiased)approach for estimating energy savings
appropriate.Since they are stipulated and,by agreement,
from efficiency programs is to measure the difference between the
fixed during the period for which savings are reported(e.g., energy use of facilities(e.g.,houses)participating in a program(the
first year or lifetime),deemed savings can help alleviate some "treatment group")and that of a similar comparison group of non-
of the guesswork in program planning and design;in effect,
participating facilities(the"control group")during the same period
they minimize one type of risk by providing certainty.
of time.The two generic categories of control group approaches are
However,deemed savings can result in another form of
randomized controlled trials and quasi-experimental methods:
risk if not properly developed and applied—overestimates Randomized controlled trials(RCTs).In an RCT,a study population
or underestimates of savings if the projects or products do (e.g.,single-family houses in Seattle that have electric heat)is
not perform as expected.This can occur,for example,if the defined and randomly assigned to either the treatment group
deemed savings value is incorrectly calculated or the deemed or the control group. Energy use(consumption)data must be
savings value was simply applied to the wrong type of application. collected for all of the project sites in the treatment and control
group in order to estimate energy savings.The energy savings
Measurement-based approaches are more appropriate estimate is then calculated by comparing the difference between
for more complex efficiency projects or for project with the measured energy use(or preferably the difference between
significant savings variability(i.e.,those with a significant the measured change in energy use)of the treatment house-
amount of savings,or"risky"savings,or with no history or holds and the energy use of the control households during the
analysis or metering on which to base a deemed savings value). same period.
Measurement-based approaches are also more rigorous than
• Quasi-experimental methods.Unlike RCTs,with quasi-exper-
deemed savings approaches and involve site data collection imental methods the assignment of the control group is not
during the period of evaluation for at least the most critical totally random.Thus,quasi-experimental methods, relative to
variables.These approaches add to evaluation costs but may RCTs,often suffer from selection bias and may produce biased
provide more accurate savings values. estimates of energy savings. However,because of the difficulty
Also,deemed savings can be used together with some and costs of conducting RCTs,quasi-experimental approaches
monitoring of one or two key parameters in an engineering are more common than RCTs,with perhaps the most common
calculation. For example,in a high-efficiency motor program, being the"pre-post"approach.With this approach,sites in the
actual operating hours could be monitored over a full work treatment group after they were enrolled in the program are
cycle.This approach is consistent with IPMVP Option A,which compared with the same sites'historical energy use prior to
is described in Chapter 4. program enrollment. In effect,this means that each site in the
treatment group is its own nonrandom control group.
December 2012 www.seeaction.energy.gov 3-6

With these approaches,statistical analyses are conducted on the It is not unusual for combinations of these approaches to be used.
energy use data(typically collected from the meter data reported on For example,rigorous randomized controlled trials may be used
monthly utility bills)and other important independent variable data every three years,with self-reported or deemed NTG ratios used for
(e.g.,weather)for those in the control and treatment groups.These the other program years.More information about determining net
approaches are primarily used for programs with relatively homog- savings is provided in Chapter 5.
enous participants and measures,when project-specific analyses are
not required or practical,but could,at least in theory,be considered
for every type of program.Example applications are large-scale THE COUNTERFACTUAL SCENARIO
weatherization programs and residential behavior-based programs.
A counterfactual analysis occurs when a person modifies
3.4.3 Approaches for Determining Net Energy a factual antecedent(a thing or event that existed before
and Demand Savings or logically precedes another)and then assesses the conse-
For the noncontrol group approaches where gross savings are quences of that modification.A person may imagine how
determined,the difference between net and gross savings is an outcome could have turned out differently if the factual
specified as a net-to-gross(NTG)ratio.The following common situation,or what led to it,did not occur.This may seem
approaches are used to determine net savings: daunting,but for energy efficiency impact evaluations,this is
• Stipulated net-to-gross ratios.These are ratios that are simply defining what the energy use(or demand,emissions,
multiplied by the gross savings to obtain an estimate of net number of jobs,etc.)would have been if the program had not
savings and are based on historical studies of similar programs. been implemented.
Sources of stipulations can cover a wide range,from simply
using"negotiated guesses"to historical values to structured The fact that energy and demand savings,as well as related
expert judgment panels.This is the least expensive approach. non-energy benefits,from efficiency efforts cannot be directly
• Self-reporting surveys and enhanced self-reporting surveys. measured results in analyses based on a counterfactual scenario.
Enhanced surveys include interviews and documentation It is counterfactual because savings are not measured,but
review and analysis.These are moderately expensive approaches. rather estimated to varying degrees of accuracy by comparing
energy consumption after a program is implemented(the
• Panel of trade allies.A significant number of trade allies reporting period)with what is assumed to have been the
provide information on their recent projects,whether the energy consumption(and demand)in the absence of the
projects are in the subject program or not,to assess the project program(the baseline or the counterfactual scenario).
program's impact on incented and nonincented program The baseline and reporting period energy use and demand
energy efficiency measures. are compared and adjusted so that only program effects are
• Large-scale consumption data analysis approaches(ran- considered when determining savings.These adjustments are
domized controlled trial methods and quasi-experimental a major part of the evaluation process and can vary from one
methods).When a control group of nonparticipants is used, program type to another and from one evaluation approach
the savings indicated are"net"of free riders and participant to another.
spillover.These are discussed in Chapter 4.
• Cross-sectional studies.These studies are comparisons of
market share of targeted technologies or behaviors between a
baseline area not served by the program and the area served
by the program.
• Top-down evaluations.These evaluations use state, regional,
or national data at a sector level to assess the extent to which
markets for energy-efficient products and services have been
affected by programs.
3-7 www.seeaction.energy.gov December 2012

Chapter 3: Notes
11 In many administrator organizations,these internal evaluations are
called"measurement and verification"or just"verification"and are
conducted by the same team that implements the programs.
12 Realization rate is used in several contexts for comparing one
savings estimate with another.The primary and most meaningful
application is the ratio of evaluated gross savings to claimed gross
savings(versus comparing net and gross savings estimates,which is
best defined with a net-to-gross ratio). Basis for the ratio not being
1.0 can include several considerations such as the following:(1)
adjustments for data errors,(2)differences in implemented measure
counts as a result of verification activities,and/or(3)other differ-
ences revealed through the evaluation process,such as with respect
to baseline assumptions.
December 2012 www.seeaction.energy.gov 3-8

Chapter 4
Calculating Energy Savings
As discussed in Chapter 3, there is no direct way of measuring energy savings, because one cannot measure the absence
of energy use. However, the absence of energy use (i.e., savings) can be estimated. Within the efficiency evaluation
industry there are three generic classifications of savings determination approaches, and the following sections describe
each of them in more detail than what was summarized in earlier chapters: measurement and verification, deemed
savings, and large-scale consumption data analysis (with the use of control groups).These descriptions are intended to
be overviews,with additional information sources referenced in Appendix C.The last section of this chapter describes
some criteria for selecting an approach. Supporting information in Chapter 5 provides information on determining net
savings. Chapter 7 provides information on other impact evaluation topics related to calculating energy savings (and
demand and non-energy benefits), and Chapter 8 discusses planning evaluation efforts.
4.1 MEASUREMENT AND VERIFICATION extensively as a program evaluation approach,primarily for"custom
APPROACH projects"whose savings are dependent on the technologies applied
Measurement and verification(M&V)is the determination of gross and/or the situations in which they are applied. For program evalua-
energy savings at individual sites or projects using one or more tion,M&V involves the following activities:
methods that can involve measurements in combination with • Selecting a representative sample of projects in a specific
engineering calculations,statistical analyses,and/or computer efficiency program,although in some programs all of the
simulation modeling. projects may be selected for M&V(a census)
Measurement and verification is a project-based approach to • Determining the savings of each project in the sample,which is
determining savings. Its genesis was in the efficiency performance done with M&V activities consisting of the following:
contracting industry,starting in the 1970s.With performance • Development of a M&V plan
contracting,a contractor implements an efficiency project for a client — Meter calibration and installation and,for long-term mea-
via a contractual arrangement that includes a savings(performance) surements,maintenance
guarantee or an arrangement that payment to the contractor is — Data gathering and screening
dependent on the savings achieved.These arrangements required — Computations of savings with measured data
"measurement and verification"to determine what level of savings — Quality assurance reviews and reporting
were being achieved at the client's facility. By the early 1990s,the
growth of the performance contracting industry was constrained, Applying the sample projects'savings to the entire population
(i.e.,the program).
in part,by the lack of robust methodologies for verifying project
savings,which,in turn,restricted the ability of the project finance
industry to participate in the market.To this end,the efficiency
industry(represented by the National Association of Energy Service UNIFORM METHODS PROJECT
Companies)14 worked with the U.S.Department of Energy(DOE),the
American Society of Heating, Refrigerating and Air-Conditioning A source of more detailed impact evaluation information is
Engineers(ASHRAE),and other stakeholders in the mid-90s to DOE's Uniform Methods Project(UMP),13 which provides
develop guidelines,which became the basis for the International model evaluation plans for specific energy efficiency
Performance Measurement and Verification Protocol(IPMVP),15 measures and project categories.These UMP documents
described below. contain additional information and specific examples that
As publicly funded efficiency programs became more prevalent in apply the concepts presented in this guide and include
the 1990s,the energy efficiency industry adopted M&V techniques examples of the three impact evaluation approaches
and concepts into the growing number of evaluations that were presented in the following sections.
being conducted for efficiency programs.Thus,M&V is now used
4-1 www.seeaction.energy.gov December 2012

FIELD INSPECTIONS OF ENERGY With M&V,energy(and/or demand)savings are determined for a
EFFICIENCY MEASURES project by comparing energy use(and/or demand)before and after
implementation of the energy efficiency measures.Thus,the following
Not all of the evaluation approaches described in this chapter fundamental algorithm applies for energy(and demand)savings:
require field inspections,but typically there are some physical Energysavings= Baseline energyuse (Reporting period
assessments for at least a sample of the individual projects in g ( )—( p g
energy use)±(Baseline adjustments)
a program(i.e.,field activities).As part of a broader verification
process,field inspections ensure that the measures installed Baseline energy use:the energy consumption that would have
meet appropriate specifications and that the projects included occurred without implementation of the energy efficiency
in a program have the potential to generate savings.This activity.When discussed in terms of specific projects,where
potential to generate savings can be verified through observa- energy saving is the metric of interest,it is sometimes called
tion,inspections,and spot or short-term metering conducted preinstallation energy use.
immediately before and after project installation.These field
• Reporting period energy use:the energy consumption that
activities can also be conducted at regular intervals during the
occurs within the time frame following implementation of
reporting period to verify a project's continued potential to
an energy efficiency activity during which savings are to be
generate savings.The field activities are an inherent part of
determined.When discussed in terms of specific projects,it is
the data collection aspects of the M&V approach,though they
sometimes called postinstallation energy use.
may be considered"add-ons"to the other approaches.
• Baseline adjustments:factors that modify baseline energy or
demand values to account for independent variables(such as
M&V also includes all field activities dedicated to collecting site weather)that influence energy use.These adjustments account
information,including equipment counts,observations of field for conditions in the reporting period that are different from
conditions,building occupant or operator interviews,measurements the conditions during the baseline period but are not a result
of parameters,and metering and monitoring. of the project activity.They distinguish properly determined
savings from a simple comparison of energy use before and
The industry's primary M&V resource is the IPMVP,which is an after implementation of a program.By accounting for independent
international end-use,energy efficiency M&V guidance document. variables that are,or are not,beyond the control of the program
It provides a framework for conducting M&V,and most important, implementer or energy consumer,the adjustments term brings
defines four M&V options that are used in the efficiency industry. energy use in the two time periods to the same set of condi-
The options involve metering of all relevant parameters,metering tions.Common examples of adjustment include the following:
of key parameters,energy use(billing)data regression analysis, — Weather corrections(e.g., if the program involves heating
and/or computer simulation.Complementing the IPMVP are the or air-conditioning systems in buildings)
Federal Energy Management Program(FEMP)M&V Guidelines.16 — Occupancy levels and hours(e.g., if the program involves
These guidelines,and companion technical notes,provide more lighting retrofits in hotels or office buildings)
details on M&V for specific measure and technology applications. — Production levels e.( g.,if the program involves energy
A third important M&V resource is ASHRAE Guideline 14(2002): efficiency improvements in factories).
Measurement of Demand and Energy Savings.17 This guideline
provides technical detail on subjects such as metering.A new One of the other considerations for adjustments is unique to
version of Guideline 14 is expected to be available in early 2013. program evaluation. In almost all bilateral performance contracts
between a contractor and its client,the baseline is considered to be
The following subsections provide material mostly from the 2010 whatever"exists in the facility"before the efficiency measures are
version of the IPMVP(EVO 10000-1:2010).They cover the basic implemented. However,in many programs using public or energy
M&V concept(algorithm)for calculating gross energy savings at the consumer funds,the baseline may be defined as common practice or
project level and introduce the four IPMVP Options—A,B,C,and as required by a code or standard(in order to not give credit for what
D—using the descriptions found in the IPMVP. would have been done without the project activity). In this situation,
the savings calculated from an end-use consumer perspective may
December 2012 www.seeaction.energy.gov 4-2

be different from the savings determined for a government agency The magnitude of such interactive effects,if significant,should be
or regulatory body.Thus,a single project may involve two different considered,and a method developed to estimate them under the
sets of savings calculations—one that supports the contract between savings determination process.
the energy services contractor and the customer(using a baseline
of existing conditions in the customer facility),and a second that The four IPMVP options(A, B,C,and D)provide a flexible set of
supports the customer's claim to a utility rebate(using a baseline of methods for evaluating project energy/demand savings.Having
current codes and standards).See Chapter 7 for more discussion of four options provides a range of methods for determining energy/
the nuances of selecting baselines. demand savings with varying levels of savings certainty and cost.
A particular option is chosen based on the specific features of each
4.1.1 Measurement and Verification Approaches project,including the following:
If M&V is a part of the evaluation process,at least some M&V details . Energy efficiency measure technologies employed and the end
will need to be specified in the evaluation planning documents,and uses in which they are applied
if sampling is used,a basis for selecting the sample of specific project
• Complexity, particularly in terms of interactive effects with
sites at which M&V activities will take place will be needed.In addition,
multiple measures and energy-using systems
as M&V is a project-specific approach to evaluation,each project
evaluated will need to have a project-specific M&V plan.There are • Potential for changes in key factors during the baseline and/or
two types of project-specific M&V plans: prescriptive method plans reporting periods
and generic method plans. • Uncertainty of the project savings as compared to the value of
• Prescriptive method plans.For project types with significant project savings
M&V"experience"and well-understood determinants of • Value of understanding the performance of the measures
savings(e.g., lighting and motor retrofits),there are established (e.g.,for a new technology).
M&V procedures,example plans,and standardized algorithms.
The FEMP M&V Guidelines contain prescriptive approaches The options differ in their approach to the level,duration,and type
for documenting savings for several common energy efficiency of baseline and reporting period measurements.The options also
measures,as does ASHRAE Guideline 14.The DOE LIMP materials differ in terms of measurement boundaries:
also include several standardized approaches to documenting • Measurement boundaries with Options A and B are made at
savings.The UMP documents do so in the context of M&V as the end use,system level(e.g.,lighting,HVAC).
an evaluation approach and are thus more germane to the . Measurement boundaries with Options C and D are at the
subject of this guide than the FEMP and ASHRAE documents whole-building or whole-facility level.
which are more specific.
• Generic method plans.There are conceptual approaches applicable Additionally,they differ in terms of type of measurements and
to a variety of project types for which prescriptive M&V methods their duration:
are not available(e.g.,comprehensive building retrofits and • Option A involves using a combination of both stipulations and
industrial energy efficiency measures).The FEMP and ASHRAE measurements of the key factors needed to calculate savings
guidelines contain several generic M&V approaches. in engineering models. Data collection tends to involve either
spot-measurements or short-term measurements.18
One of the other important aspects of M&V is defining a measurement Options B and C involve using spot,short-term,or continuous
boundary.The measurement boundary might be a single piece of measurements in engineering models(Option B)or regression
equipment(e.g.,the replaced motor in a factory),a system(e.g.,the analyses(Option C).
entire lighting system retrofitted in a commercial building),or the
whole facility(e.g.,a home that has undergone a complete retrofit). • Option D may include spot,short-term,or continuous measure-
Any energy effects occurring beyond the measurement boundary ments to calibrate computer simulation models.
are called"interactive effects."A typical interactive effect is the The four generic M&V options are summarized in Table 4.1.
decrease in air-conditioning requirements or increase in space heat-
ing requirements that can result from a lighting retrofit,which by its
nature reduces the amount of heat produced by a lighting system.
4-3 www.seeaction.energy.gov December 2012

TABLE 4.1: IPMVP Options Summary
IPMVP OPTION
CALCULATED
A.Retrofit Isolation:Key Parameter Measurement Engineering calculation of baseline A lighting retrofit in which power draw is
Savings are determined by field measurement of and reporting period energy from: the key performance parameter that is
the key performance parameter(s),which define the • Short-term or continuous measured periodically.Estimate operating
energy use of the energy conservation measures measurement of key hours of the lights based on building
(ECMs)affected system(s)and/or the success of operating parameter(s); schedules and occupant behavior.
the project. estimated values
Measurement frequency ranges from short-term to • Routine and nonroutine
continuous,depending on the expected variations adjustments as required.
in the measured parameter and the length of the
reporting period.
Parameters not selected for field measuring are
estimated.Estimates can be based on historical
data,manufacturer's specification,or engineering
judgment. Documentation of the source or
justification of the estimated parameter is required.
The plausible savings error arising from estimation
rather than measurement is evaluated.
B.Retrofit Isolation:All Parameter Measurement Short-term or continuous Application of a variable-speed drive and
Savings are determined by field measurement of the measurement of baseline and controls to a motor to adjust pump flow.
energy use of the ECM-affected system. reporting period energy,and/or Measure electric power with a kW meter
engineering computations using installed on the electrical supply to the
Measurement frequency ranges from short-term to measurements of proxies of energy motor,which reads the power every minute.
continuous,depending on the expected variations in use.Routine and nonroutine In the baseline period,this meter is in place
the savings and length of the reporting period. adjustments as required. for a week to verify constant loading.The
meter is in place throughout the reporting
period to track variations in power use.
C.Whole Facility Analysis of whole facility baseline Multifaceted energy management program
Savings are determined by measuring energy use and reporting period(utility) affecting many systems in a facility. Measure
at the whole-facility or subfacility level. meter data. energy use with the gas and electric utility
Routine adjustments as required, meters for a 12-month baseline period and
Continuous measurements of the entire facility's throughout the reporting period.
energy use are taken throughout the reporting using techniques such a simple
comparison or regression analysis.
period.
Nonroutine adjustments as
required.
D.Calibrated Simulation Energy use simulation,calibrated Multifaceted energy management
Savings are determined through simulation of the with hourly or monthly utility program affecting many systems in a
energy use of the whole facility or a subfacility. billing data.(Energy end-use facility but where no meter existed in
metering may be used to help the baseline period.
Simulation routines are demonstrated to adequately refine input data.)
model actual energy performance measured in the Energy use measurements,after
facility. installation of gas and electric meters,
are used to calibrate a simulation.
This option usually requires considerable
skill in calibrated simulation. Baseline energy use,determined using the
calibrated simulation,is compared to a
simulation of reporting period energy use.
Source:Efficiency Valuation Organization(EVO).International Performance Measurement and Verification Protocol.(2010).IPMVP,EVO 10000-1:2010.www.evo-world.org.
December 2012 www.seeaction.energy.gov 4-4

4.1.2 M&V Option A: Retrofit Isolation—Key from a simple lighting retrofit(less complex)may be more accurately
Parameter Measurement determined with Option A than could the savings from a chiller
Option A involves project-or system-level M&V assessments in retrofit(more complex).
which the savings associated with a particular project can be isolated
at the"end-use"(e.g.,ventilation system,lighting system)level.With Also true with Options A and B is that measurement of all end-use
this option, key performance parameters or operational parameters equipment or systems may not be required if statistically valid
can be measured during the baseline and reporting periods.However, sampling is used. For example,the operating hours for a selected
some parameters are stipulated rather than measured.This level of group of lighting fixtures and the power draw from a subset of
verification may suffice for types of projects in which a single parameter representative constant-load motors may be metered.
represents a significant portion of the savings uncertainty.
Savings determinations under Option A can be less costly than
Under Option A,energy and demand savings are calculated using under other options because the cost of measuring one or two
"engineering models.' These models are essentially groups of equations parameters is usually less than measuring all of the parameters.
defining energy use as a function of various inputs—often simple However,because some stipulation is allowed under this option,
spreadsheet models—and involve the development of estimates of care is needed to review the engineering design and installation to
energy and demand savings based on the following: ensure that the stipulations are realistic,applicable,and achievable
• Assumptions concerning operating characteristics of the (i.e.,the equipment can truly perform as assumed).This can be done
equipment or facilities in which the equipment is installed, through"desk reviews"of data but is more reliably done as part of
which are informed by measurements(from spot to continuous). the verification process,where site inspections check the efficiency
measure characteristics and collect site data.At defined intervals
Examples are power draws(wattage)of light fixtures or fan
during the reporting period,the installation can be reinspected to
motors and efficiencies of air conditioners(kWh/ton)and
verify the equipment's continued existence and its proper operation
heaters(Btu out/Btu in).
and maintenance.Such reinspections will ensure continuation of the
• Assumptions about how often the equipment is operated or potential to generate predicted savings and validate stipulations and
what load it serves. Examples are operating hours of lights or prior savings estimates.
fixed-speed fans and air-conditioning loads(tons)or heater
loads(Btu). 4.1.3 M&V Option B: Retrofit Isolation—All
Parameter Measurement
The most straightforward application of engineering models involves Option B,as with Option A,involves project or system-level(end-use)
using savings algorithms that calculate energy use for the subject M&V assessments with performance and operational parameters
end use(e.g.,cooling system or lighting system).Savings are then measured at the component or system level.Option B also involves
estimated by changing the model parameters that are affected by procedures for verification activities that are the same as Option A.
program participation.With Option A,at least one of the key model In addition,savings calculations,as with Option A,involve the use of
parameters must be measured.The parameters not measured are engineering models. However,unlike Option A,Option B does not
stipulated based on assumptions or analysis of facility historical data allow stipulations of any major factors that would have a significant
or manufacturer's data on the affected baseline and/or project equip- influence on energy or demand savings.
ment.It is appropriate to use a stipulated factor only if supporting
data demonstrate that its value is not subject to fluctuation over Thus,Option B requires additional and often longer-term measurements
the term of analysis and it is demonstrably applicable to the project. compared to Option A.These include measurements of both equip-
ment operating characteristics(as may be required under Option A)
This option and Option B are best applied to programs that involve and relevant performance factors(which may not be required under
retrofitting equipment or replacing failed equipment with efficient Option A).Commonly measured parameters include operating hours
models.All end-use technologies can be verified using Option A or B; for lighting and HVAC equipment,wattage for lighting and HVAC
however,the validity of this option is considered inversely propor- equipment,and flow rates and pressure for various compressed-air
tional to the complexity of the measure and the variability of its applications.Spot or short-term measurements may be sufficient
savings(e.g.,Option A is not a very reliable option for energy to characterize the baseline condition.Short-term or continuous
management system retrofits that involve complex building, measurements of one or more parameters take place after project
environment,user,and operator interactions).Thus,the savings installation to determine energy use during the reporting period.
4-5 www.seeaction.energy.gov December 2012

All end-use technologies can be verified with Option B,but determining use Options B,C,or D as needed).For programs targeting integrated
energy savings using Option B can be more difficult than doing so whole-building approaches to energy efficiency,utility bill analysis
with Option A.And,as noted above for Option A,the difficulty and can be used to statistically evaluate persistence.One useful tool for
cost increase as measurement complexity and savings variability this purpose is the U.S. Environmental Protection Agency's(EPA's)
increase.The savings,however,are typically more reliable than ENERGY STAR' Portfolio Manager(see sidebar on the next page).
those determined with Option A.
It should be noted that the term billing analysis is often used generically
4.1.4 M&V Option C:Whole-Facility Analyses to describe any analytic methodology used to determine project or
Option C involves use of whole-building meters or submeters to program energy savings based on the use of the energy consumption
assess the energy performance of a building or facility.These meters data contained in consumer billing data.It compares billing data
are typically the ones used for utility billing,although other meters, from program participants over a period of time before the energy
if properly calibrated,can also be used.With this option,energy efficiency measures are installed at customer sites to billing data for
consumption from the baseline period is compared with energy a comparable period of time afterward. If used to describe a project-
consumption data(usually derived from energy bills)from the based measurement and verification approach,it is equivalent to
reporting period.Option C also involves procedures for verification the IPMVP Option C:Whole Facility Analysis.If billing analysis is used to
activities that are the same as Option A. describe a program-based evaluation approach,it is comparable to
the large-scale consumption data analysis approach,which involves
Whole-building or facility-level metered data are evaluated using billing data from both participants and nonparticipants(control group).
techniques ranging from simple bill data comparisons to multivariate
regression analysis.Option C regression methods can be powerful 4.1.5 M&V Option D: Calibrated Simulation
tools for determining savings,while simple bill comparison methods Option D involves calibrated computer simulation models of systems,
are strongly discouraged.The latter approach does not account for system components,or whole-facility(usually residential or commercial
independent variables such as weather. buildings)energy consumption to determine project energy savings.
While in theory simulation can involve the use of any computer
For the regression analyses to be accurate,all substantive explana- analysis tools,such as spreadsheets,these calibrated simulations are
tory(independent)variables that affect energy consumption need to typically associated with complex building analysis tools that model
be monitored during the performance period.Substantive variables heating,cooling,lighting,ventilation,and other energy flows as well
may include weather,occupancy schedules,industrial throughput, as water use and sometimes onsite air emissions. Examples of such
control set points,and operating schedules.Most applications programs are DOE-2 and EnergyPlus.19 The quality of the savings
of Option C require at least 9 to 12 months of monthly baseline estimate depends on how well the simulation models are calibrated
(preinstallation)meter data and at least 9 to 12 months of monthly and how well they reflect actual performance.
data from the reporting period(postinstallation).
All end-use technologies can be verified with Option C. However,this INTERACTIVE FACTORS AND LEAKAGE
option is intended for projects in which savings are expected to be
large enough to be distinguishable from the random or unexplained Interactive effects are those that an energy efficiency
energy variations normally found at the level of the whole-facility measure has on energy use in a facility,but which are
meter.The larger the savings,or the smaller the unexplained varia- indirectly associated with the measure.For example,
tions in the baseline consumption,the easier it will be to identify reduction in lighting loads through an energy-efficient
savings. In addition,the longer the period of savings analysis after lighting retrofit will reduce air conditioning and/or increase
project installation,the less significant the impact of short-term heating requirements,since there is less heat generated
unexplained variations.Typically,savings should be more than 10% by the energy-efficient lights.When energy efficiency
of the baseline energy use so that they can be separated from the programs have interactive effects beyond a single building
"noise"in baseline data. and start to affect energy supply and distribution systems,
Option C is the most common form of M&V for some multi-measure there can be implications for calculating avoided emissions
and other related co-benefits. In this situation of wide-scale
building energy efficiency retrofits in the performance contracting
interactive effects,the term leakage is used.
industry(although ESCOs use Option A as much as possible and then
December 2012 www.seeaction.energy.gov 4-6

U.S. EPA!S PORTFOLIO MANAGER 4.2 DEEMED SAVINGS APPROACH
Deemed savings values,also called stipulated savings values,are
One tool that can be used to analyze facility utility billing estimates of energy or demand savings for a single unit of an
meter data is U.S. EPA's Portfolio Manager(PM).Over installed energy efficiency measure that(1)has been developed from
300,000 buildings have been benchmarked with PM,which data sources(such as prior metering studies)and analytical methods
provides a consistent framework and metric that building that are widely considered acceptable for the measure and purpose,
energy managers can use to track,measure,and monitor and(2)is applicable to the situation being evaluated. Individual
whole-building energy use. PM employs a methodology parameters or calculation methods can also be deemed;for example,
that is consistent with IPMVP Option C.It aggregates all the effective useful life of a measure or the annual operating hours of
meter data from a building so that performance changes can light fixtures in an elementary school classroom.Common sources
be assessed at the whole-facility level.Savings are deter- of deemed savings values are previous evaluations and studies that
mined at the building level to promote system-wide energy involved actual measurements and analyses. Deemed savings values
reductions.Additionally,because the PM approach combines are used for both planning and evaluation purposes.
multiple meters,it accounts for differences among fuel
types.This is done by converting site meter data into source Deemed savings are used to stipulate savings values for projects with
energy consumption.www.energystar.gov/portfoliomanager. well-known and documented savings values.Examples are energy-
efficient appliances such as washing machines,computer equipment,
and refrigerators,and lighting retrofit projects with well-understood
Typically,reporting period energy use data are compared with the operating hours. Many performance contracting projects document
baseline computer simulation energy use prediction(using reporting their savings with deemed savings,and it is also a popular evaluation
period independent variable values)to determine energy savings, approach for many efficiency programs because of both the relatively
although simulation results(reporting period)to simulation results low cost of using deemed savings and the certainty of savings values
(baseline)are also used to determine savings. that all parties can rely on for their own purposes.
Models are often calibrated by comparing simulation results with The use of deemed values in a savings calculation is thus essentially
historical data to ensure that the models have accurately captured an agreement between the involved parties to an evaluation to
the operating characteristics of the building. Manufacturer's data, accept a stipulated value or a set of assumptions for use in determining
spot measurements,or short-term measurements may be collected the baseline or reporting period energy consumption.With the
to characterize baseline and reporting period conditions and operating deemed savings approach,it is increasingly common to hold the
schedules.The collected data serve to link the simulation inputs to stipulated value constant regardless of what the actual value is during
actual operating conditions.The model calibration is accomplished the term of the evaluation. If certain requirements are met(e.g.,
by comparing simulation results with end-use or whole-building verification of installation and performance,satisfactory commis-
data.Whole-building models usually require at least 9 to 12 months sioning results,and sufficient equipment or system maintenance),
of preinstallation data for baseline model calibration. However,these
models are sometimes calibrated with only reporting period data so
that they can be used with new construction projects for which no BUILDING ENERGY
baseline data exist. SIMULATION PROGRAMS
Any end-use technology can be verified with Option D if the drop in
consumption is larger than the associated simulation modeling error. For about 40 years,engineers and scientists have been
This option can be used in cases in which there is a high degree of developing computerized models that describe how the
interaction among installed energy systems or the measurement energy use of buildings changes in response to independent
of individual component savings is difficult.Option D is commonly variables such as weather.The sophistication and complexity
used with new construction energy efficiency programs,where the of these models is quite varied.To learn about some of the
baseline is typically modeled using standard practice or building building simulation models that are publicly available,visit
code requirements to define what would have occurred without the http://simulationresearch.lbl.gov/resources.
efficiency activity.
4-7 www.seeaction.energy.gov December 2012

the project savings are considered to be confirmed.The stipulated to be applied to calculated savings(e.g.,net-to-gross values),source
savings for each verified installed project are then summed to documentation,specified assumptions,and other relevant material
generate a program savings value. Installation might be verified by to support the calculation of measure and program savings as well as
physical inspection of a sample of projects or perhaps just an audit the application of such values and algorithms in appropriate applica-
of receipts.Savings can also be verified for"persistence"with tions. For example,a value for operating hours in an elementary
periodic inspections that verify that the retrofits are still in place classroom with no summer hours should not be applied to a high
and functioning. school classroom with summer sessions.
A variant of deemed savings is the deemed savings calculation, As of the date of this guide's publication,there are approximately
which is an agreed-to(stipulated)set of engineering algorithm(s) 17 TRMs in use across the United States.These include state and
used to calculate the energy and/or demand savings associated with regional TRMs,which are listed in Appendix C.In a recent SEE Action
an installed energy efficiency measure.These calculations are devel- report on TRMs,21 it was shown that these resources are very valu-
oped from common practice that is widely considered acceptable able,but there are a wide variation in methodologies for estimating
for the subject measure and its specific application. It may include savings and the actual values.Some TRMs include information based
stipulated assumptions for one or more parameters in the algorithm, on prior year evaluations including,in some cases,rigorous metering
but typically it requires users to input data associated with the actual and analysis,and thus these TRMs contain robust(reliable)savings
installed measure into the algorithm(s). values. Many others have values based on analyses(e.g.,using com-
puter simulations or engineering algorithms),with consideration in
The use of deemed savings is quite popular for evaluations of energy their calculations of waste heat factors,in-service rates,and partial
consumer-funded programs.The following is from a recent survey of load factors.The transparency and level of detail regarding methods
evaluation practices in the United States: and assumptions also ranges from substantial to minimal.
We found that nearly all states(36 states,86%)use some type Thus,as would be expected when using any assumptions or stipulated
of deemed values in the evaluation framework.In terms of values in an analysis,caution should be used to understand the
what types of values are"deemed,"we found 35 states(97%of sources of such values and ensure that the assumptions that went
those responding to this question)deem savings amounts for into determining a value are applicable to the situation(e.g.,mea-
particular measures,32 states(89%)deem the"lifetime"over sures,measure delivery mechanism,facility types)being evaluated.
which to claim savings for particular measures,and 20 states Deemed values,if used,should be based on reliable,traceable,and
(65%)deem free-ridership or net-to-gross factors. documented sources of information,such as the following:
We also inquired about the source of the deemed values used • Standard tables from recognized sources that indicate the
by the states.It appears that there is a lot of"borrowing" power consumption(wattage)of certain pieces of equipment
going on within the industry.Twenty-six states(70%)cite the that are being replaced or installed as part of a project
use of sources or databases from other states. In nine states, (e.g.,lighting fixture wattage tables)
the utilities develop and file certain key deemed values,and in • Manufacturer's specifications
two states,the Commission is responsible for developing the • Building occupancy schedules
deemed values.In most states(28 states,80%),the results of . Maintenance logs.
their own in-state evaluations are used to modify and update
deemed values over time.20 In addition,it is good practice to have an ongoing process in place
to assess the validity of deemed savings values,such as an annual
Deemed savings values and deemed savings calculations are usually or biennial process to update TRMs. In particular,check to see if the
documented in a database in formats from spreadsheets to online assumptions used to determine deemed savings values are valid in
searchable databases.A term of art for such databases is technical the years after their initial determination(e.g.,the applicability and
reference manuals(TRMs).These are resource documents that validity of assumed code requirements or standard practices).
include energy efficiency measure information used in program plan-
ning and energy efficiency program reporting. It can include savings When using deemed values,it is important to realize that technologies
values for measures,measure life information,hourly load shapes of alone do not save energy;it is how they are used that saves energy.
savings,engineering algorithms to calculate savings,impact factors Therefore,a deemed energy savings value depends on how and
December 2012 www.seeaction.energy.gov 4-8

where a technology is placed into use.For example,a low-wattage and quasi-experimental methods(QEMs).Both involve the use of
lamp's savings are totally dependent on its operating hours.Such control groups,but RCTs provide less biased and typically more
a lamp installed in a closet will save much less energy than one precise results than QEMs,although they may require more effort to
installed in a kitchen. implement. For efficiency programs,RCTs and QEMs are particularly
useful for programs in which there are a relatively large number of
The example of the residential lamp raises the issue of"granularity" participants with similar characteristics(e.g.,low-income,single-fam-
of the deemed savings values.In that example,if an average house- ily houses in a particular city).Currently,these methods are primarily
hold's annual operating hours were used to estimate savings,the used for evaluations of residential behavior-based programs and
result would be underestimated savings if lamps were only installed whole-house retrofits and weatherization programs.
in high-use areas(kitchens)and overestimated savings if lamps were
only installed in low-use areas(closets).Thus,the value of deemed 4.3.1 Randomized Controlled Trial (RCT) Methods
savings depends not only on the validity of the value used,but on In an RCT,first a study population is defined(such as homes eligible for
whether the value is applied correctly(i.e.,it must be based on the a residential weatherization program or a behavior-based program in
use conditions as well as the technology). a particular city).Then the study population is randomly assigned to
either the treatment group or control group.23 Energy use data must
In summary,sources of stipulated values must be documented in be collected for all facilities in both the treatment group and control
the evaluation plan. Even when stipulated values are used in place group in order to estimate energy savings. Measured energy use
of measurements,equipment installation and proper operation typically comes from utility meter or billing data,often in hourly or
are still verified. Properly used,stipulations can be very useful for monthly increments.
program planning purposes and can reduce M&V costs,create
certainty,and simplify evaluation procedures. Improperly used, The energy savings estimate is then calculated in one of two ways
they can give evaluation results an inappropriate aura of authority. (options):(1)by comparing the difference between the measured
Deciding whether parameters could be stipulated requires that users energy use of the treatment group households24 and the energy use
understand how they will affect savings,judge their effect on the of the control households during the evaluation study period,or(2)
uncertainty of results,and balance the costs,risks,and goals of the by comparing the energy use reduction(i.e.,the change in use from
program being evaluated. before the study period to after the study period)between the treat-
ment households and the control households.Thus,savings from
4.3 LARGE-SCALE CONSUMPTION DATA the program are essentially the difference in changes in energy use
ANALYSIS (RANDOMIZED CONTROLLED (positive or negative)between the treatment group and the control
TRIALS AND QUASI-EXPERIMENTAL group. Between the two options listed above,the second is prefera-
METHODS) ble so that the differences between the control group and treatment
As indicated in Chapter 3,a reliable approach for estimating energy group with respect to their differences in energy use before and after
savings from efficiency programs is to measure the difference the program implementation(i.e.,differences in differences)
between the energy use of those facilities(e.g.,houses)participating are determined.
in a program(the"treatment group")and the energy use of a
comparison group of nonparticipating facilities(the"control group') Random assignment is a defining aspect of RCTs. It means that each
that are similar to those in the participant group during the same facility(e.g.,household)in the potential program participant population
period of time.The difference between the energy use ofthe treatment is randomly assigned to either the control or treatment group(i.e.,
group and the control group facilities can be attributed to those that will be in the program)based on a random probability,
three sources: as opposed to being assigned to one group or the other based
1. The true impact of the program on some characteristic of the facility or participant(e.g.,location,
energy use,or willingness to sign up for the program).Randomization
2. Preexisting differences between households in the treatment thus eliminates preexisting differences that are observable(e.g.,
and control group,which is called "bias"or"selection bias" energy use or household floor area)as well as differences that are
3. Inherent randomness.22 typically unobservable(e.g.,attitudes regarding energy conservation,
number of occupants,expected future energy use,and occupant
There are two generic types of large-scale consumption data age)unless surveyed.Thus, because of this random assignment,
analyses discussed in this guide: randomized controlled trials(RCTs) an RCT control group is an ideal comparison group:it is statistically
4-9 www.seeaction.energy.gov December 2012

identical to the treatment group in that there are no pre-existing 4.3.2 Quasi-Experimental Methods (QEMs)
differences between the two groups,which means that selection Other evaluation design methods that use nonrandomized control
bias is eliminated. Randomized controlled trials can also be used groups are called quasi-experimental methods.With these methods,
for various program enrollment options,including opt-in,opt-out, the control group is not randomly assigned.Thus,quasi-experimental
and a randomized encouragement design that does not restrict methods often suffer from selection bias that may produce biased
program participation.25 estimates of energy savings—sometimes very biased results that
result in unreliable savings estimates. However,in some specific
It is worth pointing out two net savings aspects that RCTs address: cases in which RCTs are not feasible,quasi-experimental approaches
free riders and participant spillover.This is one of the main benefits can still provide reliable results(especially compared with deemed
of an RCT over traditional evaluation methods.Thus,if net savings savings values and certain M&V approaches).
are defined for a program evaluation as the gross savings plus
consideration of free riders and participant spillover, RCTs deliver The sections that follow provide brief discussions of some common
estimates of net savings. QEMs.
RCTs address the free-rider concern because the treatment and Pre-Post Energy Use Method
control groups each contain the same number of free riders through Probably the most common quasi-experimental method is to
the process of random assignment to the treatment group or the compare the energy use of participants in the treatment group
control groups.When the two groups are compared,the energy after they were enrolled in the program to the same participants'his-
savings from the free riders in the control group cancel out the torical energy use prior to program enrollment. In effect,this means
energy savings from the free riders in the treatment group,and the that each participant in the treatment group is its own nonrandom
resulting estimate of program energy savings is an unbiased estimate control group.This is called a pre-post within subjects,or interrupted
of the savings caused by the program(the true program savings). time series design analysis.
Participant spillover is also automatically captured by an RCT design.
The challenge in using this method is that there are many other
An RCT design also addresses rebound effects or take-back during factors(independent variables)that may influence energy use before,
the study period,which can occur if consumers increase energy use during,and after the program that are not captured with this
(e.g.,as a result of a new device's improved efficiency).Rebound method, resulting in biased savings estimates.Some of these factors,
is sometimes,although not often,a consideration in determining such as differences in weather or number of occupants,can be
savings persistence. Rebound effects after the study period can be measured and reliably accounted for in the analysis. However,other
accounted for with an RCT if the data collection is continued for the factors are less easily observed and/or accounted for. For example,
time under consideration. the economy could worsen,leading households and businesses to
decrease energy use(even if there were no program);prices for
However,free riders and participant spillover are not the only factors energy can change,affecting energy use;and cultural norms could
differentiating gross and net savings.The following are some other change—perhaps,say a pop culture icon could suddenly decide to
net-to-gross considerations that are not addressed by either RCTs or advocate for energy efficiency.27
the QEMs about to be discussed:
• Nonparticipant spillover issues,in which a program influences To minimize bias when using the pre-post calculation method,it is
the energy use of non-program participants(although there are necessary to include a regression analysis that discerns and controls
some specialized techniques that can determine nonparticipant for the impact of other influences(e.g.,economic recession)that
spillover)26 may affect energy use over time compared with the impact of the
• Natural gas-related changes in energy use if only electricity efficiency program.Simple comparison before and after energy use
consumption is measured is not acceptable.
• Long-term market effects such as changes in efficiency product Matched Control Group Method
pricing and availability(unless control and treatment groups If it is not possible to create a randomized control group,then savings
are maintained over long periods)or the influence of a pro- estimates could be calculated by constructing a nonrandom control
gram on establishing an energy efficiency measure as common group made up of participants that are as similar to the treatment
practice or part of a code or standard. group as possible.The challenge with a matched control group
December 2012 www.seeaction.energy.gov 4-10

method is that participants and their facilities have both observable and other statistical and econometric methods,to analyze
characteristics(e.g.,level of energy use,ZIP code,presence of measured energy use data and to control for variations in
central air conditioning)that could potentially be matched,as well independent variables.
as unobservable characteristics(e.g.,energy attitudes or propensity
to opt in to an energy efficiency program)that are harder or With regression analyses,an equation or group of equations that
impossible to match. model the relationship between the dependent variable and one
or more important independent variables is defined. Dependent
Variation in Adoption variables are those that are modeled to be influenced by the inde-
The variation in adoption approach takes advantage of variation in pendent variables. Independent variables are the variables that are
the timing of program adoption in order to compare the energy use not influenced by other variables in the model,and are assumed to
of participating facilities that optin to the program with the energy affect or determine the dependent variables,and are thus the inputs
use of facilities that have not yet opted in but will ultimately opt in to an analysis.Independent variables include both the variable(s)of
at a later point.It relies on the assumption that in any given month, interest(e.g.,a variable that indicates which customers experience
participants that have already opted in and participants that will opt a critical peak pricing event during which hours)as well as control
in soon share the same observable and nonobservable characteris- variables that seek to observe and account for other factors that may
tics.Thus,this method creates a control group that is very similar to influence the dependent variable(e.g.,the average temperature).
the treatment group over time in both observable and unobservable Which independent variables are relevant to calculating energy sav-
characteristics,and therefore is likely to result in less bias than ings?Often,this is decided by common sense,experience,program
matched control or pre-post methods. characteristics,or budget considerations(with respect to how many
variables can be measured and tracked),but it also can be determined
Regression Discontinuity Method through field experiments and statistical tests. For weather data,the
Among the quasi-experimental methods,regression discontinu- most common independent control variable,there is a wide range of
ity typically yields the most unbiased estimate of energy savings. public and private data sources.
However,it is also the most complicated method,as it requires
knowledge of econometric models and often requires field condi- In the case of energy efficiency analyses,the output of a regression
tions that allow the evaluator to use this analytic technique. analysis is a coefficient that estimates the effect of independent
Therefore,it is neither common nor always practical.This method variables(e.g.,a program)on the dependent variable(energy or
works if the eligibility requirement for households to participate in demand consumption and/or savings).The analysis itself is done
a program is a cutoff value of a characteristic that varies within the with a computer model,which can be anything from a spreadsheet
population. For example,households at or above a cutoff energy tool to sophisticated proprietary statistical modeling software.
consumption value of 900 kWh per month might be eligible to par-
ticipate in a behavior-based efficiency program,while those below 4.4 SELECTING AN ENERGY SAVINGS
900 kWh are ineligible. In this case,the households that are just EVALUATION APPROACH
below 900 kWh per month are probably very similar to those that Selecting an evaluation approach is tied to objectives of the program
are just above 900 kWh per month.Thus,the idea is to use a group being evaluated,the scale of the program,evaluation budget and
of households right below the usage cutoff level as the control group resources,and specific aspects of the measures and participants
and compare changes in their energy use to households right above in the program.The following subsections describe situations in
the usage cutoff level as the treatment group.This method assumes which each of the three generic impact approaches discussed in the
that the program impact is constant over all ranges of the eligibility proceeding sections are applicable. More information on planning
requirement variable that are used in the estimation(e.g.,that the evaluation activities is included in Chapter 8.
impact is the same for households at all levels of energy use). In
addition,regression discontinuity relies on the eligibility requirement One criterion that is applicable across all of the approaches is evaluator
being strictly enforced. experience and expertise.Thus,a common requirement for selecting
an approach is that the evaluator has expertise with the approach
4.3.3 Analysis Techniques selected.A related requirement is that the resources required for
All of the methods described above(RCTs and QEMs)use one of that approach—such as the data,time,and budget—are available.
a number of different analysis techniques,including regressions
4-11 www.seeaction.energy.gov December 2012

QUALITY
The M&V approach is used for almost any type of program that
involves retrofits or new construction projects.While a census of The impact evaluation approaches described in this guide
projects can be used with the M&V approach,it is generally applied are based on new and unique analysis of energy and demand
to only a sample of projects in a program.This is because the M&V savings.Sometimes,however,there is documentation
approach tends to be more expensive on a per-project basis than that indicates energy and demand savings were calculated
the other two approaches. In general,the M&V approach is applied independently of the subject impact evaluation,such as
when the other approaches are not applicable,such as when there information prepared by implementers and/or administra-
are no deemed savings values available that are applicable to the tors.Although such documentation was not necessarily
given measure or combination of measures,or when per-project prepared per pre-determined evaluation requirements,it
results(savings)are needed.An example is a performance-contract- may be sufficient for meeting the evaluation objectives.
ing program with multiple contractors. Using existing documentation in combination with quality
assurance guidelines(QAGs)can save significant costs for the
Because the selection of the M&V approach is contingent on which of program sponsor—and perhaps encourage participation in
the four M&V options is selected,Table 4.2 summarizes some selec- the program if a portion of evaluation costs is borne by the
tion criteria for each M&V option.Table 4.3 indicates factors that participants.Essentially,a QAG can help determine whether
affect the cost of implementing each option.Table 4.4 and Figure 4.1, indicated savings,and the assumptions and rigor used to
both from the 2010 version of the IPMVP,indicate key project char- prepare the documentation,can be used in place of a new
acteristics that are better for the different options and a flowchart evaluation effort.
summarizing the selection of M&V options,respectively.These tables
and the figure are included as the last pages of this chapter.
4.4.2 Deemed Savings Approach • The uncertainty associated with savings estimates is low and/or
The deemed savings approach is most commonly used for programs the risk of under-(or over-)estimating savings is low.That is,
that involve simple new construction,or for retrofit energy efficiency the project's likelihood of success is high.
measures with well-defined applications and savings calculations • Documented,reliable,and applicable per-measure stipulated
that have been verified with data. Examples might be a residential values are available and applicable to the measure installation
second-refrigerator recycling program or a CFL giveaway for residen- circumstances.
tial utility customers. In each of these two examples,an assumption • The primary goal of the evaluation is to conduct field inspections
would be made about the baseline and energy savings,as well as for all or a sample of projects to ensure they are properly installed
perhaps the life of the measures(e.g., 15 years for a refrigerator and have the potential to generate savings(rather than having
and 10,000 hours for a CFL).The deemed savings values would have rigorously determined energy savings).
to be defined for specific applications(e.g.,second refrigerators in
single-family housing)and program delivery mechanisms(point of Assessing a few key aspects of the project can help in making decisions
sale or direct install).The deemed savings values might also have a about whether to use deemed savings or deemed calculations.
"vintage"in that they are valid for certain years of a program or for Uncertainty in predicted savings,and the degree to which individual
certain vintage baseline equipment. parameters contribute to overall uncertainty,should be carefully
considered in deciding whether to use stipulations.The"rules of
In general,the deemed savings approach is most applicable when all thumb"are as follows:
(or at least most)of the following are true: The most certain, predictable parameters can be estimated
• There are limited evaluation resources. and stipulated without significantly reducing the quality of the
• The projects involve simple energy efficiency measures with evaluation results.
well-understood savings mechanisms that have been verified • Stipulating parameters that represent a small degree of uncer-
with data,and are not subject to significant variation in savings tainty in the predicted result and a small amount of savings will
due to changes in independent variables.The stipulated values do not produce significant uncertainty concerns.
not significantly increase the uncertainty of the evaluation metrics.
December 2012 www.seeaction.energy.gov 4-12

• Parameters should be measured when savings and prediction
uncertainty are both large.
• Even if savings are high, but uncertainty of predicted savings is
low,full measurement may not be necessary for M&V purposes.
4.4.3 Large-Scale Consumption Data
Analysis Approach
These approaches are used for programs that have many participants
that share many common characteristics,such as single-family
detached homes in a particular community with residents of similar
economic demographics.These can be equipment retrofits,new
construction,or behavior-based programs. Because of the require-
ment for a large amount of data,this approach is almost always used
with residential programs,such as a weatherization program with
thousands of homes being retrofitted with a variety of measures
(e.g.,insulation,weather stripping,low-flow showerheads,and CFLs).
In general,the large-scale consumption data analysis approach is
most applicable to programs that meet most(if not all)of the
following criteria:
• Participation is well defined(i.e.,the specific consumers or
facilities that participate in the program are known).
• The program has a relatively large number of participants
(i.e.,probably more than 100).
• At least one year's worth of baseline and reporting period
energy consumption data are available for both the treatment
group and the control group. If an RCT method is used,
a shorter data period may be adequate.
• If an RCT method is not used,then there are observable
similarities between participants,or relatively homogenous
subgroups of participants can be formed with similar facility
and energy efficiency measure characteristics.
• Either the program design is such that the target population
for the program can be randomly divided into participants and
nonparticipants(for randomized controlled trial methods),or
sufficient data about the characteristics of the participants are
available for selecting an appropriate control group for quasi-
experimental methods.This is a particularly important criterion
with respect to control group members,as they typically do not
have an incentive to provide data to the evaluator.
• Independent,third-party evaluators select the treatment and
control group,rather than program implementers,to minimize
the potential for"gaming."
4-13 www.seeaction.energy.gov December 2012

TABLE 4.2: Applications for Each IPMVP M&V Option
OPTION OPTION OPTIONRetrofit Isolation- Key Retrofit Isolation- Whole Facility Calibrated Simulation
Parameter Measurement All Parameters is best . . applied
is best applied where: Measurement
is best applied
• The magnitude of savings The project involves simple • The project is complex • New construction projects
is low for the entire project equipment replacements . predicted savings are large are involved
or for the portion of the Energy savings values (typically greater than 10%) • Energy savings values per
project to which Option A per individual measure compared to the recorded measure are desired
is applied are desired energy use
• Option C tools cannot
• The project is simple . Interactive effects are to • Energy savings values cost-effectively evaluate
with limited independent be ignored or are stipulated per individual measure particular measures
variables and unknowns using estimating methods are not needed
• Complex baseline adjust-
• The risk of not achieving . Independent variables • Interactive effects are to ments are anticipated
savings is low are not complex be included
• Baseline measurement
• Interactive effects are to • Independent variables that data do not exist or are
be ignored or are stipulated affect energy use are not prohibitively expensive
using estimating methods complex or excessively to collect
difficult to monitor
Source:Efficiency Valuation Organization(EVO).(2010).International Performance Measurement and Verification Protocol.IPMVP,EVO 10000-1:2010.www.evo-world.org.
TABLE 4.3: Factors Affecting the Cost of Each M&V Option
OPTION A OPTION B
Retrofit Isolation- Key Retrofit Isolation- OP • OPTION
Parameter Measurement All Parameters Whole Facility Calibrated Simulation
Measurement i
• Number of measurement • Number of points • Number of meters to be • Number of meters to be
points and independent installed and/or analyzed installed and/or analyzed
• Complexity of deriving variables measured . Number of independent • Number of independent
the stipulation • Complexity of measure- variables used in models variables used in models
• Frequency of postretrofit ment system • Effort required for
inspections • Length of time measure- calibration of models
ment system maintained
• Frequency of postretrofit
inspections
December 2012 www.seeaction.energy.gov 4-14

TABLE 4.4: Key Project Characteristics and Applicability for Different M&V Options
ECIVI PROJECT CHARACTERISTIC SUGGESTED OPTION
Need to assess ECMs individually X X X
Need to assess only total facility performance X X
Expected savings less than 10%of utility meter X X X
Multiple ECMs X X X
Significance of some energy driving variable is unclear X X X
Interactive effects of ECM are significant or unmeasurable X X
Many future changes expected with measurement boundary X X
Long-term performance assessment needed X X
Baseline data not available X
Nontechnical persons must understand reports X X X
Metering skill available X X
Computer simulation skill available X
Experience reading utility bills and performing regression analysis available X
Source:Efficiency Valuation Organization(EVO).(2010).International Performance Measurement and Verification Protocol.IPMVP,EVO 10000-1:2010.www.evo-world.org.In this
table,ECM stands for energy conservation measure,which is equivalent to energy efficiency measure as used in this guide.
4-15 www.seeaction.energy.gov December 2012

FIGURE 4.1: M&V option selection flowchart
START
ECM FACILITY
PERFORMANCE PERFORMANCE
Measure Facility or
ECM Performance?
Able to isolate NO
ECM with meter(s)? YES Expected NO
Savings>10%?
YES
Need to separately
assess each ECM?
Need full NO YES
performance
demonstration?
NO
YES Analysis of
If main meter
Install isolation Install isolation meters data Simulate
meters for all for key parameters, system or
parameters and assess interactive facility
assess interactive effects, and estimate
effects well known parameters
Obtain
calibration data
Missing baseline YES
or reporting Calibrate
period data? simulation
Missing baseline or NO
reporting period data? Simulate with
and without
YES ECM(s)
NO
OPTION B OPTION
WholeRetrofit Isolation: Retrofit Isolation:
All Parameter Key Parameter . -.
Measurement Measurement Simulation
Source:Efficiency Valuation Organization(EVO).(2010).International Performance Measurement and Verification Protocol.IPMVP,EVO 10000-1:2010.www.evo-world.org.In this
table,ECM stands for energy conservation measure,which is equivalent to energy efficiency measure as used in this guide.
December 2012 www.seeaction.energy.gov 4-16

Chapter 4: Notes
13"Uniform Methods Project."(2012). U.S.Department of Energy. 21 State and Local Energy Efficiency Action Network.(June 2011).
www.eere.energy.gov/deployment/ump.html. Scoping Study to Evaluate Feasibility of National Databases for
14 NAESCO.www.naesco.org. EM&V Documents and Measure Savings. Prepared by Jayaweera,1;
Haeri, H.; Lee,A.; Bergen,S.; Kan,C.;Velonis,A.;Gurin,C.;Visser,M.;
15 Efficiency Valuation Organization(EVO). International Performance Grant,A.; Buckman,A.;The Cadmus Group Inc.www.eere.energy.
Measurement and Verification Protocol(IPMVP).(multiple dates). ov seeaction dfs emvsco in databasefeasibilit
g / /p / p g_ Y•pdf.
www.evo-world.org.The Efficiency Valuation Organization(EVO)is a
non-profit organization that also offers a Certified M&V Professional 22 Randomness can be a factor in these analyses. For example,it
Certificate Program. could be that for the chosen population,during the time interval
the energy use was monitored,the households in the treatment
16 Federal Energy Management Program(FEMP).(April 2008). group randomly happened to lower their energy use at around the
M&V Guidelines:Measurement and Verification for Federal Energy same time that the program started.The precision of an estimate of
Projects.wwwl.eere.energy.gov/femp/pdfs/mv_guidelines.pdf. energy savings quantifies the effect of this inherent randomness and
Other related M&V documents can be found at http://mnv.lbl.gov/ allows us to decide whether it is a problem or not. Bias and preci-
keyMnVDocs/femp. sion,as well as the general subject of certainty of savings estimates,
17 American Society of Heating, Refrigerating and Air-Conditioning is discussed in Section 7.4.
Engineers(ASHRAE).(2002).Guideline 14:Measurement of Demand 23The control and treatment groups could contain equal sizes of
and Energy Savings.www.ashrae.org.Also ASHRAE.(2010). households,or the control group could be bigger or smaller than
Performance Measurement Protocols for Commercial Buildings. the treatment group. It is only necessary to keep a control group
www.ashrae.org/standards-research--technology/special--project- that is sufficiently large to yield statistical significance of the savings
activities.These documents provide a standardized,consistent set of estimate.
protocols for facilitating the comparison of measured energy,water,
and indoor quality performance of commercial buildings. 24 Because households are the most common example,the rest of
this section uses households as the example unit of analysis.
18 Spot measurements are one-time measurements,such as the
power draw of a motor.Short-time measurements might take place 25 State and Local Energy Efficiency Action Network.(May
for a week or two,such as to determine the operating hours of lights 2012).Evaluation, Measurement and Verification(EM&V)of
in an office.Continuous measurements,as the name implies,involve Residential Behavior-Based Energy Efficiency Programs:Issues and
measuring key factors such as power consumption or outdoor Recommendations. Prepared by Todd,A.;Stuart, E.;Schiller,S.;
temperature throughout the term of the evaluation,which may be Goldman,C.; Lawrence Berkeley National Laboratory.wwwl.eere.
years. Utility billing meters provide continuous measurements and energy.gov/seeaction/pdfs/emv_behaviorbased_eeprograms.pdf.
are the typical measurements used with IPMVP Option C.The use A complete source of information on use of RCTs(and QEMs)for
of longer-term measurements can help identify under-performing evaluation of energy efficiency programs,particularly behavior-based
energy efficiency projects,which in turn can lead to improvements in residential programs. Much of the text in this section is from that
their performance. report.
19 DOE-2 information can be found at http://appsi.eere.energy.gov/ 26 it is possible to explicitly conduct an experiment to determine
build ings/tools_directory/alpha_list.cfm and EnergyPlus information the spillover effects. For example,an experiment could observe the
can be found at http://appsl.eere.energy.gov/buildings/energyplus/ impacts of the intervention for households in the experiment as well
energyplus_about.cfm.An extensive list of building energy simula- as others and compare the impacts in two or more communities.
tion programs can be found at:http://appsl.eere.energy.gov This type of experiment is often used in medicine and epidemiology.
build ings/tools_directory/alpha_list.cfm. 27 However,in some programs,such as critical peak pricing or critical
20 Kushler, M.; Nowak,S.;Witte, P.(February 2012).A National peak rebates,a pre-post method may be less biased.This is because
Survey of State Policies and Practices for the Evaluation of Ratepayer- with such programs the experimental factor(e.g.,the call to curtail
Funded Energy Efficiency Programs.American Council for an Energy- electricity use or a price signal)can be presented repeatedly so that
Efficient Economy(ACEEE). Report Number U122.www.aceee.org/ the difference between the energy use when it is present and when
research-report/u122. it is not present is observable.
4-17 www.seeaction.energy.gov December 2012

Chapter 5
Determining Net Energy Savings
This section describes an important, and perhaps the most controversial of, impact evaluation topics—the determination
and use of net savings. Net (energy and/or demand) savings are the changes in energy consumption or demand that are
attributable to an energy efficiency program.The net-to-gross (NTG) ratio is the portion (it can be less than or greater
than 1.0) of gross savings (those that occur irrespective of whether they are caused by the program or not)that are
attributed to the program being evaluated.
Determining net savings involves separating out the impacts that To help understand these terms,the following are some of the
are a result of influences other than the program being evaluated, related definitions:
such as consumer self-motivation or effects of other programs.The Free rider:a program participant who would have implemented
controversy of net savings can be summarized as follows: the program's measure(s)or practice(s)in the absence of the
Given the range of influences on consumers'energy consumption—and program.Free riders can be(1)total,in which the participant's
the complexity in separating out both short-term and long-term activity would have completely replicated the program's
market effects caused by the subject programs(and other pro-
intended actions; (2)partial,in which the participant's activity
grams)—attributing changes to one cause(i.e.,a particular program) would have partially replicated the program's actions;or(3)
or another can be quite complex. deferred, in which the participant's activity would have partially
or completely replicated the program's actions,but at a future
This controversy is compounded by a lack of consensus by energy time beyond the program's time frame.
efficiency policymakers and regulators as to which short-term and Spillover(participant and non-participant):the reduction in
long-term market influences and effects should be considered when energy consumption and/or demand caused by the presence
determining net savings and the role of net savings in program of an energy efficiency program,beyond the program-related
design,implementation,and "crediting"of savings to program gross savings of the participants and without financial or
administrators. technical assistance from the program.There can be participant
and/or non-participant spillover.Participant spillover is the
The following subsections start with classic energy efficiency industry additional energy savings that occur when a program participant
definitions of net savings,including the commonly used factors that independently installs incremental energy efficiency measures
differentiate net from gross savings,and a discussion of the uses of or applies energy-saving practices after having participated in
net savings determinations.Next is a description of the approaches the efficiency program as a result of the program's influence.
used to determine net savings.The final subsection discusses some Non-participant spillover refers to energy savings that occur
of the issues and nuances of net savings. when a program non-participant installs energy efficiency
5.1 DEFINITIONS AND USES OF NET SAVINGS measures or applies energy savings practices as a result of a
program's influence.Sometimes the term free drivers is used
5.1.1 Defining Net Savings for those who have spillover effects.
The energy efficiency community agrees on the basic definition of Market effect:a change in the structure of a market or the
net savings:the total change in energy use(and/or demand)that is behavior of participants in a market that is reflective of an
attributable to an energy efficiency program.However,as noted in increase(or decrease)in the adoption of energy-efficient
a recent regional scoping paper on net savings,the operational products,services,or practices and is causally related to market
definition of net savings—essentially what factors are considered intervention(s)(e.g.,programs). Examples of market effects
when determining net savings—is not unanimously agreed to,with include increased levels of awareness of energy-efficient tech-
different factors being applied in different programs and jurisdictions.28 nologies among customers and suppliers,increased availability
Factors that cause the difference between net and gross savings, of efficient technologies through retail channels,reduced prices
implicitly or explicitly,include free ridership,participant and non- for efficient models,build-out of efficient model lines,and—the
participant spillover,and induced market effects.These factors may end goal—increased market share for efficient goods,services,
be considered in how a baseline is defined(e.g.,common practice) and design practices.
and/or in adjustments to gross savings values.
December 2012 www.seeaction.energy.gov 5-1

In terms of how different jurisdictions define net savings,and which FIGURE 5.1: Single program evaluation components
of the above factors are included,a 2012 American Council for an
Energy-Efficient Economy study indicated the following:29
"I'lllql
We asked states what they used when they report their energy
savings results,and found that 21 states(50%)said they reported
net savings,12 states(29%)said gross savings,and 9 states(21%)
said they report both(or use one or the other for different FREE
purposes).We explored the net savings issue in a little more RIDERS
detail,and asked whether states made specific adjustments XXAOS
for free riders and spillover.Interestingly,while 28 states(67%)
indicated they make an adjustment for free riders,only 17 states
SPILLOVER
(44%)make an adjustment for free drivers/spillover. MARKET EFFECTS
It is important to recognize that the study survey did not specify any
particular definition of what qualifies as net or gross savings.Rather,
they allowed states to categorize their own approach.
From a summary paper of evaluation practices,the traditional eval-
FIGURE 5.2: Multiple program evaluation overlaps
uation approach to net savings assumes a measurement boundary
around a program,as shown in Figure 5.1.30 The participant savings,
which are the documented savings from the efficiency measures
installed through the program's transactions with the customer,
are shown in the shaded circle.Any of those savings attributableFREI
to free riders are subtracted out(inner circle),leaving the shaded
donut of net participant savings. If there are spillover savings,either Or I& FREE
F RIDERS
from additional measures installed by participants without program
incentives or from measures installed by non-participants who were
indirectly influenced by participants'actions,these may be added
to the total. Finally,as the programs begin to affect the market(e.g., RIDERS
by inducing retailers to sell only efficient equipment in response to
market demand),there may be additional savings(i.e.,the market Fr FREE SPILLOVER Ai
RIDERS
effects).These additional savings would expand the shaded area of
countable program savings. SPILLOVER
FREE
MARKET EFFEC_[S _A RIDERS
However,as also pointed out in the above-referenced paper,with
the evolution of integrated portfolios of programs targeted at broad
and deep savings goals,the situation looks more like Figure 5.2.Note
that the overlaps may,in practice,be even greater than this simpli-
fied diagram would suggest.Often,the participant savings overlap FREE V
between program offerings,and one program's participants may i RIDERS
be another program's free riders.The overlaps can also occur over
time,with one program's influence extending to the next generation SPILLOVER
of programs.The point of the diagram is to suggest that the simple MARKET EFFECTS
evaluation strategy of drawing a boundary around each program,
over a single program cycle,will encounter the problems of multiple
program influences that can occur before,during,and after the
Source:Mahone,D.;Hall,N.(August 2010)."Pursuit of Aggressive Energy Savings
Targets—Aligning Program Design and Evaluation."2010ACEEESummer Study Pro-
subject program's implementation.This can lead to biased estimates ceedings;August 15-20,2010,Pacific Grove,California.Washington,D.C.:American
Of net savings,either higher or lower than actual. Councilforan Energy-Efficient Economy.http://0eep.org/uploads/EMV%20Foruml
Steering%20Committee%2ONotesIDMahone_ACEEE Paper for SC Notes.pdf.
5-2 www.seeaction.energy.gov December 2012

Implications of these complexities are discussed after the following The following considerations about net savings are important to
subsections on the uses of net savings and the methods used to define for an evaluation effort,but also can be a potential source of
determine net savings. controversy due to the different perspectives and objectives of those
who will use the evaluation results:
5.1.2 Importance and Uses of Net Savings
1. Which factors should be included in the definition of net
Generally speaking,net savings are of most interest for regulated
savings(e.g.,free ridership,spillover,and/or long-term market
government and utility programs.In these cases,the responsible
art e. effects/market transformation)
party( g.,a city council or utility regulator)wants to know if the use
of public or energy consumer-funded programs are actually having 2. Whether common methods of determining net savings do or
an influence,and thus their efficiency investments are"wise! That is, do not properly account for these factors
"Are the programs of interest providing incremental benefits,or do 3. Whether and how net savings can be used in a retrospective
the benefits result from some other influences?"For example,in the manner,in particular for determining attribution and if an
case of programs funded by utility customers(the most common administrator has or has not met its savings goals.
situation where the issue of free riders comes up),why would one
group of consumers subsidize the efficiency actions of other consum- These first two issues are discussed after the following subsection on
ers that would have taken the action anyway?Or,as another example, methods used to determine net savings.The last issue above is simi-
the environmental benefits of energy efficiency programs are usually lar to the issue of how savings estimates are applied,as discussed in
considered valid only if they are additional to naturally occurring Section 8.3.5.
efficiency activities(i.e.,based on some version of net savings). 5.2 APPROACHES FOR DETERMINING
In contrast,there are other situations where gross savings are of the NET SAVINGS
most interest.For example,private sector energy efficiency performance Before describing the methods used to determine net savings,it
contracts are a case where gross energy savings are the primary is important to understand that beyond just defining what factors
concern.Also,from a"bottom line"resource and environmental are considered when defining net savings,the actual calculation
perspective,it may not be relevant what exactly caused a change of net energy and demand savings can be more of an art than a
in energy consumption,only that it is occurring or will continue to science.Essentially,one is attempting to separate out the influence
occur.Table 5.1 summarizes different"public"audiences and their of a particular energy efficiency program(or portfolio)from all the
current uses of net savings estimates. other influences—such as self-motivation,energy prices,and other
From the net savings scoping paper referenced earlier in this section, efficiency programs—that determine participant and non-participant
as well as other sources,there are at least four uses of net savings behavior and decisions.With the increasing"push"for energy
that do not appear to be controversial,paraphrased below: efficiency by utilities and governments at the local,state,and
•
Using net savings, prospectively,for program planning and national level and by private groups and large companies,it can be
design (e.g.,for setting consumer incentive levels). quite difficult to separate out how one particular program among
all this activity influences consumer decisions about whether,when,
• Assessing the degree to which programs cause a reduction in and to what degree to adopt efficiency actions.
energy use and demand—with net savings as one of numerous
measures that should be given serious consideration in the As indicated in Chapter 3,net savings evaluation methods can be
assessment of program success. categorized as follows:
• Obtaining insight into how the market is changing and trans- Stipulated net-to-gross(NTG)ratios.These ratios are multiplied
forming over time by tracking net savings across program by the gross savings to obtain an estimate of net savings and
years and determining the extent to which free ridership and are based on historical studies of similar programs.Sources of
spillover rates have changed during the period—and potentially stipulations can cover a wide range,from simply using"negoti-
using this insight for defining when,and how,to implement ated guesses",to historical values(perhaps based on prior year
a program exit strategy. NTG studies),to structured expert judgment panels.
• Gaining a better understanding of how the market responds Self-reporting surveys.Information is reported by participants
to the program and how to use the information to inform and non-participants without independent verification or
modifications to program design, including how to define review. Respondents are simply asked if they would have
eligibility and target marketing.
December 2012 www.seeaction.energy.gov 5-3

TABLE 5.1: Audiences for Net Energy Savings
ESTABLISHED
AUDIENCE EMERGING USE OF NET SAVINGS ESTIMATES
AUDIENCE
Energy efficiency Established(in some areas, • Assess if program achieved savings goals
program administrators legislators and advocacy • Identify strong and weak areas of program design and redesign
and planners;energy groups are emerging program accordingly
regulators;legislators; audiences)
• Apply strong program designs for other products,in other
advocacy groups
jurisdictions
• Adjust payments to/funding of programs based on goal
achievement
• Determine if the ratepayer/taxpayer funds are being spent
cost-effectively and wisely
• Define program administrator financial incentives and/or cost
recovery levels
Air regulators Emerging • Will apply emission factors to energy savings to estimate
greenhouse gas and other avoided emission reductions
• Assess degree to which efficiency programs have achieved
greenhouse gas reduction and other avoided emissions targets
Source:Modified version of information provided in NMR Group,Inc.;Research Into Action,Inc.(November 2010)."Net Savings Scoping Paper."Submitted
to Northeast Energy Efficiency Partnerships:Evaluation,Measurement and Verification Forum.http://neep.org/uploads/EMV%20Forum/EMV9lo20Products/
FINAL%20Net%20Savings%20Scoping%20Paper%2011-13-10.pdf.
undertaken the action promoted by the program on their own monetary compensation for information on their recent(e.g.,
without the incentive(free ridership).Then,they are asked last 50)projects.Details requested would include manufacturer,
whether they had undertaken additional energy efficiency efficiency levels,size,price,installation date,installation location
actions(purchased products or made a behavioral change) (ZIP code),whether the project was eligible for energy program
as a result of their participation in the program(participant incentives,whether energy incentives were received,whether
spillover).Through non-participant surveys, respondents are any other incentives were received(e.g.,tax credits or manu-
asked if they had recently undertaken energy efficiency actions facturer rebates),and an assessment of the program's impact
and if those actions were undertaken as a result of the utility on incented and non-incented efficiency actions.Trade allies
program(s)(non-participant spillover). would include both program participants and non-participants.
• Enhanced self-reporting surveys.The self-reporting surveys This approach,while not currently common,can yield reliable
are combined with interviews and documentation review and information on standard market practices,and—through an
analysis.The survey instruments themselves include more ongoing annual update—provide context for tracking ongoing
enhanced batteries of questions(e.g.,would you have taken program impacts or market effects.
exactly the same action as promoted by the program or would large-scale consumption data analysis approaches(randomized
you have undertaken the action at the same time?). controlled trial methods and quasi-experimental methods).
• Panel of trade allies.A significant number of trade allies(e.g, Statistical models are used to compare energy and demand
contractors,retailers,builders,and installers)are offered patterns of a group of participants and a control group.Where
5-4 www.seeaction.energy.gov December 2012

a control group of non-participants is used,the savings indicated Each of these methods comes with its own data collection
are"net"of free riders and participant spillover. and measurement challenges. Particularly when dealing
• Cross-sectional studies.These studies consist of comparisons of with subjective indicators,such as why someone chose to
market share of targeted technologies or behaviors between a implement an energy efficiency measure,care should be taken
baseline area not served by the program and the area served by in not only obtaining the needed data but also in reporting it
the program.The main disadvantage of this type of study is diffi- with appropriate indications of the quality and reliability of
culty in obtaining quality data.Also,as energy efficiency programs results obtained from the data.
become more prevalent,finding control areas(areas without
similar program activities)is becoming exceedingly difficult. In terms of accuracy requirements,the challenge in surveying
comes from the nature of collecting both qualitative and quantitative
• Top-down evaluations(or macro-economic models).Top-down data from various participants and non-participants involved in the
evaluations use state, regional,or national data at the sector decision to install energy efficiency measures.Another uncertainty
level to assess the extent to which markets for energy-efficient challenge in surveying is the subjective nature of assigning NTG ratios
products and services have been affected by programs(See to each participant—their free ridership and participant spillover
Appendix B). "score"A participant is a"total"free rider if he or she would have
The following are some general notes on these methods: absolutely installed the exact same project at the exact same time,
at the same price,even if the program did not exist—and they
• The most commonly used methods are the survey-based self- know that.Assigning NTG ratios to individual participants is more
reporting,stipulation,and the large-scale consumption data complicated,however,in cases where the participant had multiple
analysis approaches. reasons for making the project decision,might have installed a
• Net savings values can be the output of(most)large-scale data different project,or would have installed it in two years if not for the
analysis and top-down evaluations.With the other impact program—or all of the above.Table 5.2 shows an approach that one
analysis approaches(M&V and deemed savings),the net evaluator used to define full,partial,and deferred free riders.
savings correction is calculated independently.
• All of these methods can address,to one degree of reliability The following subsections discuss the more common NTG determi-
or another,free ridership. However,the ability of the methods nation methods:surveys and stipulation of the NTG ratios.
to address spillover is limited in terms of how the method is 5.2.1 Self-Reporting Surveys
applied and whether non-participants as well as participants Surveys can be complex to design and administer. Respondents'
are included in the analyses.Market effects can only be analyzed perception and understanding of the questions is absolutely critical
if the studies are conducted over a long period of time,and if to the success of the inquiries.Surveying approaches have become
such effects are actually"looked for."When selecting a method, somewhat standard practice with the guidelines developed for
it is very important to define what factors are included in the Massachusetts and the self-report guidelines developed by the
definition of net savings and whether a selected method can California Public Utilities Commission's Energy Division.31
actually and reliably address all of the factors.
• With respect to program size and scale,the survey methods, The survey approach is the most straightforward way to estimate
stipulation,and cross-sectional studies can be used with any free ridership and spillover. It is also the lowest cost approach.It
program,regardless of the number of participants. However, does,however,have its disadvantages, regarding potential bias
the top-down and large-scale consumption data analysis and overall accuracy. For example,typical responses such as"don't
approaches can only be used with programs that have a large know,"missing data,and inconsistent answers are very hard to
numbers of participants.This is because the models need large address without additional data collection.While there are ways to
amounts of data to provide reliable results. improve survey quality(e.g.,using techniques like adding consistency
• In terms of timing of net savings analyses,it is preferable to check questions and adjusting the individual's estimate accordingly),
analyze a long period of time to address spillover effects. the accuracy of simple self-reports is typically marginal.
Conversely,the free ridership reporting accuracy is probably One of the elements that should be addressed in surveys is self-
highest when the inquiry is made as close as possible to the selection bias.Self-selection bias is possible whenever the group
actual energy efficiency action. being studied has any form of control over whether to participate
December 2012 www.seeaction.energy.gov 5-5

TABLE 5.2: Example of Assigning Free Ridership Percentages
WOULDFEE- ALREAD •
IP Y INSTALLED SAME INSTALLED AD
WITHOUTRIDERSH ORDERED OR
SCORE • •
PROGRAM MEASURES
100% Yes Yes — — — —
0% No No — — — —
0% No Yes No — — —
50% No Yes Yes Yes Yes Yes
25% No Yes Yes Yes No Yes
25% No Yes Yes Yes Yes No
0% No Yes Yes Yes No No
25% No Yes Yes No Yes Yes
F12.5% No Yes Yes No No Yes
12.5% No Yes Yes No Yes No
0% No Yes Yes No No No
Source:Courtesy of The Cadmus Group,Inc.It is also safe to assign a score of 100%free ridership to those that had already installed the measure prior to receipt of rebate.
in the survey;for example, people who have strong opinions or equipment?"For a response,participants might choose between
substantial knowledge may be more willing to spend time answer- "definitely would have "probably would have "probably would
ing a survey than those who do not.Self-selection bias is related to not have,"and"definitely would not have."This use of a scale,rather
sample selection bias and can skew the results of a NTG analysis that than a yes/no response,is thought to allow greater confidence and
is not very well planned,funded,and/or executed. precision in the estimate.Based on the responses to the various
questions,each response is assigned a free-ridership score.These
Another form of survey bias is response bias:the tendency of respon- estimates are then combined(additively or multiplicatively)into an
dents to gauge their responses to conform to socially acceptable values. individual participant free-rider estimate.The participant estimates
This issue is well recognized in social sciences and is discussed in a are subsequently averaged(or assigned a weighted average based on
vast body of academic and professional literature.Another aspect of expected savings)to calculate the overall free-ridership estimate.
response bias is construct validity,which raises questions about what
the survey results actually measure.The problem stems from the 5.2.2 Enhanced Self-Reporting Surveys
fact that while survey respondents,by virtue of their participation in To improve the quality of NTG ratios drawn from self-reported survey
the program,are predisposed to efficiency,it is not clear to what extent responses,the evaluation can rely on multiple data sources for the
their responses are conditioned by the effects of the program itself. decision to install or adopt energy efficiency measures or practices.
Some common additional data sources and techniques include
Generally,the best means for implementing self-reporting surveys the following:
have involved asking a series of questions,with each question allow- personal surveys.Conducting in-person surveys is probably the
ing a scale of responses.A typical initial question asked of participants best way to qualitatively improve the quality of self-surveys.
is,"If the program had not existed,would you have installed the same
5-6 www.seeaction.energy.gov December 2012

Key participants in the decision to install efficiency measures sets the value. It is the simplest approach,but one with a high
can help determine the level of influence of the program on potential for inaccuracy relative to other approaches that involve
participants and non-participants. For commercial and gov- some level of data collection and analyses.
ernment facilities, potential interviewees include managers,
engineers,and facilities staff.Contractors,design engineers, Sources of stipulated NTG ratios include evaluations of similar programs,
and product manufacturers,distributors,and retailers can also hopefully applied to similar populations with a similar level of effi-
provide information on the influences and motivations that ciency adoption and during a time period similar to that of the program
determine the role of energy efficiency programs in the being reviewed.Other sources use historical or other information
decision-making process.When working with professionals from a wide range of sources to develop a"weight of evidence"
involved in the efficiency measure installation,individuals conclusion regarding the program's influence. For example,in a
familiar with the program and projects should conduct the three-year portfolio cycle,a stipulated NTG ratio may be used for
interviews.The interviewer should attempt to eliminate or at the second two years based on a NTG ratio determined with other
least minimize any bias they may have. approaches in the first year of the portfolio.One common approach
• Project analysis.This consists of two general types of reviews. for developing a stipulated value is to use a panel of experts that have
The first is an analysis of the barriers to project installation and relevant technology,infrastructure systems,and market experience.33
how the project addresses these barriers.A common barrier These experts are asked to estimate a baseline market share for a
is financial (project costs),so an analysis is done of a project's particular energy efficiency measure or behavior and,in some cases,
simple payback. For example,if the project has a very short forecast market share with and without the program in place.
payback period without any program-provided benefits,then it 5.2.4 Selecting a Net Savings Evaluation
may be considered as more likely to have been installed with or Approach and Timing
without the program.32 The other type of analysis is to review As mentioned in Chapter 4,selection of an evaluation approach is
any documentation the participant may have of the decision tied to the objectives of the program being evaluated(e.g.,to help
to proceed with the project.Such documentation may include understand/improve program design or to adjust savings estimates),
internal memos or feasibility studies,and can indicate the basis the scale of the program,the evaluation budget and resources,and
of the decision to proceed. specific aspects of the measures and participants in the program.
• Market data collection.Through the review of other informa- Another criterion—probably the most important one cited for these
tion resources prepared for similar programs,the survey data studies—is the cost of the net savings analysis.The lowest-budget
can be adjusted.Such resources might include analyses of approach is to use stipulated NTG ratios,followed by self-reporting
market sales and shipping patterns,studies of decisions by par- surveys and enhanced surveys,and then various cross-cutting and
ticipants and non-participants in similar programs,and market modeling approaches;although,if the data are available,the top-
assessment, potential,or effects studies. Market sales methods down evaluation approach can be quite inexpensive.One option for
rely on aggregate data on total sales of a particular technology keeping costs down while using the more sophisticated approaches
in a given jurisdiction.They compare this sales volume with a is to conduct an NTG ratio analysis every few years and stipulate NTG
baseline estimate of the volume that would have been sold in ratios for the intervening years as long as the market influences and
the absence of the program.The accuracy of these methods participants'behavior are relatively consistent.
depends on the completeness and accuracy of the sales data as
well as the validity of the baseline estimate. 5.3 ISSUES AND NUANCES OF NET SAVINGS
As noted above,within the energy efficiency industry,it is agreed
All or some of these three data sources can be combined with that net savings determinations are well used for assessing certain
written or Web-based participant and non-participant self-surveys to programmatic features such as focusing program designs to maximize
triangulate on an estimate of the free ridership and spillover. their efficacy.The net savings controversy is over(1)what factors
5.2.3 Stipulated Net-to-Gross Ratio should be included in the definition of net savings;(2)whether
This approach,although not a calculation approach,is often used. common methods of determining net savings do or do not properly
NTG ratios are stipulated in some jurisdictions when the net savings account for free ridership,participant and non-participant spillover,
value is not considered critical,or if the expense of conducting NTG and long-term market effects(i.e.,market transformation);and(3)
analyses and/or the uncertainty of the potential results are whether administrator goal achievement should be based on net
considered significant barriers. In such a situation,a regulatory body savings or gross savings.
December 2012 www.seeaction.energy.gov 5-7

As summarized in the EM&V Forum"Net Savings Scoping Paper" However,in certain regulatory environments,when free-ridership
referenced in Table 5.1,the issues can be summarized in one levels are deemed excessive,program administrators are penalized,
word—attribution: claimed savings are discredited,and programs are cancelled.34
Attribution assessment has always involved isolating the effects Beyond the application of free-ridership results,as noted in the
of the program from other influences. Increasingly,however, section above on spillover and market effects,the actual determina-
when the energy efficiency community mentions the"challenge of tion of free ridership can be difficult if the large-scale consumption
attribution"or"sorting out attribution,"it refers to the fact that data analysis approaches of randomized controlled trials and certain
reductions in end users'energy consumption can be affected quasi-experimental methods are not used(although these methods
not only by myriad efficiency programs offered by a broad range do not separate out participant spillover and free ridership from one
of sponsors,but also by economic ups and downs,changes in another and do not include non-participant spillover and long-term
energy prices,concerns about climate change,and ongoing market effects).And,as noted in the section on methods,the most
advances in technology,among other influences.This situation common method for determination of free ridership is participant
has significantly exacerbated the difficulty of establishing causa- surveys—the"self-reports."Self-reports can have problems of
tion,and,therefore,of estimating net savings,and it is likely response bias beyond the issue of whether people can state their
that this situation will persist.Because of the increased difficulty reasons for undertaking efficiency activities(or,for that matter,any
of establishing causation,some commentators in the energy behavior that humans undertake).
efficiency community believe that the net savings estimates
developed recently are less accurate than those developed in In addition,in areas with long histories of efficiency programs and
the past when there were fewer programs and messages activities and many programs operating at the same time,it may not
promoting efficiency and"being green." be possible to parse out who is a free rider and who was influenced
by the program. In effect,it may be that,in the case of transformed
The following are some brief discussions covering some specific net markets or markets being transformed,what is being measured in
savings issues that are subsets of this attribution question.These are free-ridership surveys is in fact spillover from other programs.35
presented to not necessarily provide complete answers and recom-
mendations,as the solutions tend to be jurisdiction-specific,but to 5.3.2 Spillover and Market Effects
point out these issues and suggest that these issues be considered as For many energy efficiency programs,the ultimate goal is market
part of the evaluation planning process(see Chapter 8). transformation,where"interventions"are no longer required for
the specific measure or practice because market share is such that
5.3.1 Free Ridership adoption is outstripping standard practice.Therefore,it can be a
Free-ridership issues are by no means peculiar to energy efficiency; primary evaluation objective for metrics associated with spillover
they arise in many policy areas,whenever economic agents are paid and market effects to be assessed. Unfortunately,many net savings
an incentive to do what they might have done anyway.However,few determinations only consider free-ridership levels and/or assess
issues bring about more discussion in the energy efficiency industry performance for the first year of a program's implementation—too
than free ridership. Even the use of the term itself is controversial,as short to assess market effects or long-term spillover benefits. In
the way in which it is used in the efficiency industry diverges from its addition,when assessing free ridership,a survey-based method may
classic economic definition. be asking the questions of the wrong people.Those identified as free
riders might actually be exactly the type of participants that policy-
The basic concept speaks to the prudent use of energy efficiency makers would want for a market transformation program,those who
dollars:they should be spent to encourage customers to take energy will take the action and continue to do so once the intervention is
efficiency actions that they would not otherwise take on their own. over,when the market is transformed.36 It may be that free ridership
If program dollars are spent on people who would have taken the is only low in programs(and efficiency actions)that can never result
actions anyway,without program support,then those people are in transformed markets.This may be creating a counter-incentive for
free riders,and those dollars were perhaps misspent. Evaluators are administrators to implement programs that cannot result in market
tasked with studying how much of a program's resources were spent transformation—by not counting the savings from their free riders
on free riders,and what the program savings were,net of free riders. and/or spillover—and giving high marks for programs with low free
The consequences of free-ridership measurements vary. In some ridership(i.e.,those programs where only an intervention will get
cases,the information is used to refine program plans to better target people to implement the efficiency activity).
customers and to assess progress toward market transformation.
5-8 www.seeaction.energy.gov December 2012

If overly conservative free-ridership measurements are taken,and 5.3.3 Baselines and Net Savings
if free ridership is used to penalize programs,then some program Related to determining free-ridership levels and spillover is the point
efforts may be killed prematurely before market transformation that free-ridership savings essentially raise a question about what
or ambitious levels of savings are achieved.This also relates to the the correct baseline is—what would have occurred in the absence
importance of estimating program spillover and overall market of the program or project. In short,would someone have done this
effects.As an efficiency measure is moving up the market transfor- anyway?Some baselines,such as a codes and standards baseline
mation"hump,"spillover should increase,as there are more satisfied or existing conditions baseline,may require a free-rider analysis
consumers implementing more of the efficiency actions,even after the program(project)is implemented to get participant-by-
without the interventions. Indeed,without the snowball effect of participant indications of free ridership.However,if market-based
spillover,programs have a large burden to push the measure up and "common practice"was used to define the baseline for calculating
over the hump entirely on their own.37 energy savings,the resulting estimates could require no further
adjustments for free riders,as the estimates include consideration of
Most current approaches to defining net savings do not address what typically would have been done in the absence of the efficiency
the benefits that programs have on establishing infrastructures that action.See chapter 7 for more information on baseline selection.
allow future efficiency actions(equivalent to a bus passenger being
able to be a free rider only because prior programs resulted in the This"common practice"approach to baselines can be used to
road and the buses being built and the drivers trained and hired), include consideration of free riders.What"common practice"
because program impacts are generally measured over 1-3 years, baselines do not account for is spillover or long-term market effects
rather than 5+years. Net-to-gross ratio adjustments,including (i.e.,delayed participation/spillover)created by prior programs.
those that account for spillover,typically presume that the measure However,neither does the use of other definitions of baselines.
would have been equally available at the same price had no prior Therefore,if common practice baselines(i.e.,net savings)are to
programs existed. include spillover and other market effects,additional adjustments
are required.
The following is an approach to this dilemma,paraphrased from a
paper on this subject:38 5.4 SUMMARY
An approach for policymakers would be to set market targets The above discussion on the factors associated with net savings
(e.g.,a percentage of market share)and perform market studies
determination is intended to show the complexities of net savings
that track the progress toward increased market-share.There determination.It is not intended to indicate that net savings should
is no known"tipping point"percent at which any specific not be determined or that net savings metrics do not have value.
technology will be likely to flip to the point of majority adoption As noted previously,there are many valuable uses of net savings
(market transformation). But,it is possible to make estimates determinations.
of the rate of adoption to determine whether the rate of adop- What is suggested is that the limitations of net savings determina-
tion is occurring in a manner that justifies public support,and
lion be acknowledged in the policy setting and evaluation planning
at what level.This recommendation leaves open the possibility
processes,and that,when net savings are determined,there be
that incentives may be even more important for later adopters
clear indications of what factors are and are not included in the
than for early ones.This approach allows program design
determination,and over what time frame. Building on the policy
decisions about how to adjust the program over time to be
and program implementation strategy recommendation mentioned
supported by real-time data on market progress and a clear
above,with respect to market effects and spillover,is this overall
sense of the desired direction for the market through realistic
recommendation:39
goal setting and adjusting.
With the above points in mind,another conclusion can be that Regulators could establish a series of hurdles,or tests,that
program administrators should budget for tracking of market data, a program has to pass to avoid high free ridership.The exact
such as sales volumes for specific energy-efficient equipment,price nature of the tests would vary depending on the program,but
the amount of the incentive relative to the cost of the measure
information,and market saturation indicators.This data collection
is a good general gauge.When very low incentives appear
fits under the category of market effects studies.These market
to attract a large number of participants,or net benefits to
effects studies are briefly discussed in Appendix B.
December 2012 www.seeaction.energy.gov 5-9

participants are very high,chances are the majority of partici-
pants will be free riders. Programs administrators must avoid
offering incentives for projects with very short paybacks to
participants who most likely would—or should—undertake the
project on the project's own financial merits.
[With respect to evaluation activities] program administrators
would have to monitor energy-efficient product markets closely
to see if a transformation has occurred and exit the market
when it has. Expected savings and costs of conservation mea-
sures should be revised periodically based on actual saturation
of energy-efficient products through well-designed and detailed
market effects studies. Baselines can also be adapted over time,
based on market baseline studies.
5-10 www.seeaction.energy.gov December 2012

Chapter 5: Notes
28 NMR Group,Inc.;Research Into Action, Inc.(November 2010). 35 Haeri,H.;Khawaja,M.S.(March 2012)."The Trouble With
"Net Savings Scoping Paper."Submitted to Northeast Energy Freeriders."Public Utilities Fortnightly.www.fortnightly.com/
Efficiency Partnerships:Evaluation, Measurement,and Verification fortnightly/2012/03/trouble-freeriders.
Forum. http://neep.org/uploads/EMV%20Forum/EMV%20Products/
FINAL%20Net%20Savings%20Scoping%20Paper%2011-13-10.pdf. 36 Haeri,H.;Khawaja,M.S.(March 2012)."The Trouble With
Freeriders.' Public Utilities Fortnightly.www.fortnightly.com/
29 Kushler, M.;Nowak,S.;Witte,R(February 2012).A National fortnightly/2012/03/trouble-freeriders.
Survey of State Policies and Practices for the Evaluation of Ratepayer-
Funded Energy Efficiency Programs.American Council for an Energy- 37 Mahone,D.(August 2011)."Free-Ridership as a Way to Kill
Efficient Economy(ACEEE).Report Number U122.www.aceee.org/ Programs—How Evaluation Policies Can Frustrate Efficiency Goals."
research-report/u122. Boston:International Energy Program Evaluation Conference.
www.iepec.org/20llPapersTOC/papers/054.pdf.
30 Mahone,D.; Hall, N. (August 2010)."Pursuit of Aggressive
Energy Savings Targets—Aligning Program Design and Evaluation." 38 Peters,J.;McRae,M.(August.2008)."Free-Ridership Measurement
2010 ACEEE Summer Study Proceedings;August 15-20,2010, Is Out of Sync With Program Logic...or,We've Got the Structure Built,
Pacific Grove,California.Washington, D.C.:American Council but What's Its Foundation?"2008 ACEEE Summer5tudy Proceedings.
for an Energy-Efficient Economy. http:Hneep.org/uploads/ August 17-22,2008,Pacific Grove,California.Washington, DC:
EMV%20Forum/Steering%20Committee%20Notes/DMahone_ American Council for an Energy-Efficient Economy(ACEEE).
ACEEE_Paper_for_SC_Notes.pdf. www.aceee.org/files/proceedings/2008/data/papers/5_491.pdf.
31 Rathbun,P.;Sabo,C.;Zent,B.(2003).Standardized Methods for 39 Haeri,H.;Khawaja,M.S.(March 2012)."The Trouble With
Free-Ridership and Spillover Evaluation—Task 5 Final Report(Revised). Freeriders."Public Utilities Fortnightly.www.fortnightly.com/
Prepared for National Grid,NSTAR Electric,Northeast Utilities,Unitil, fortnightly/2012/03/trouble-freeriders.
and Cape Light Compact.www.ceel.org/eval/db_pdf/297.pdf.
California Public Utilities Commission(CPUC).(2007).Guidelines for
Estimating Net-To-Gross Ratios Using the Self-Report Approaches.
CPUC Energy Division Master Evaluation Contractor Team.
ftp://ftp.cpuc.ca.gov/puc/energy/electric/energy+efficiency/
ee+workshops/se Ifreportguideli nesd eta iled_v20.pdf.
32 Simple payback is a common metric but it is complicated to apply.
For example,what one entity might consider a short payback another
might see as a long payback.Also,because the payback is usually
calculated based on energy cost savings,it ignores other benefits to
participants,such as increase in value of property,more reliability,
comfort,and others,and thus can overestimate the payback period
from the participant's perspective.
33This is called a Delphi process,which is an interactive forecasting
method using a panel of experts.
34 Mahone,D.(August 2011)."Free-Ridership as a Way to Kill
Programs—How Evaluation Policies Can Frustrate Efficiency Goals."
Boston:International Energy Program Evaluation Conference.
www.iepec.org/20llPapersTOC/papers/054.pdf.
December 2012 www.seeaction.energy.gov 5-11

Chapter 6
Calculating Avoided Air Emissions
State and federal policymakers and utility regulators are broadening the scope of efficiency programs, and thus their
evaluation, by focusing on objectives beyond energy savings. Examples of these broader objectives include reducing
the need for on-peak generation (demand response), promoting economic development through job creation, reducing
greenhouse gas emissions, and achieving a wide range of air quality and health benefits.40 Because avoided air emissions
in particular are seen as a significant benefit of energy efficiency actions throughout the United States,this chapter has
been included to provide guidance for those interested in documenting these benefits.
This chapter first features a summary of the topic of avoided emissions Historically,emission reductions from efficiency activities were usually
and end-use energy efficiency,which is followed by a discussion of only described subjectively in program evaluations as a non-quantified
special issues associated with this topic.The remainder of the chapter (non-monetized)benefit.This is changing with increasing interest in
focuses on ways in which emission factors can be calculated,with quantifying these benefits for at least two purposes:
one section covering calculation of emission factors associated with
avoided onsite fuel use and the following section covering avoided • Determining the cost-effectiveness of efficiency programs
emissions calculations for grid-connected electricity approaches.The (to possibly justify more investment in efficiency)by
second-to-last section provides brief summary comments on selecting monetizing the environmental benefits of efficiency
a calculation approach,and the last section provides further references • Supporting state claim of emissions benefits in state air
on this topic.Also included in this chapter is a discussion of issues pollution plans(e.g.,State Implementation Plans—SIPS—for
related to avoided emissions calculations,including additionality, criteria pollutants)or GHG reduction requirements(e.g.,
boundary area definitions,and the design of cap-and-trade programs. California's Assembly Bill 32:Global Warming Solutions Act42).
6.1 ENERGY EFFICIENCY AND AVOIDED
EMISSIONS SUMMARY
EFFICIENCY AS . COST-EFFECTIVE
Energy efficiency can reduce air emissions associated with the EMISSIONS REDUCTION STRATEGY
production of electricity and thermal energy from fossil fuels.Air
emissions that can be reduced by efficiency include the six commonly Energy efficiency policies and programs offer the potential to
found air pollutants(also known as the U.S.Environmental Protection achieve emissions reductions at a cost that can be lower than
Agency[EPA] "criteria pollutants"): particle pollution(often referred traditional control measures.The EPA is gaining experience
to as particulate matter or PM),ground-level ozone,carbon monox- with these potentially cost-effective strategies in rulemak-
ide,sulfur dioxide,nitrogen oxides,and lead.All of these pollutants, ings affecting the utility and other sectors. For example,
as well as others such as mercury,have geographic-specific impacts; a recent EPA modeling scenario for EPA's Mercury and Air
therefore,where the emission reduction occurs is critical to deter- Toxics Standard rule predicts that moderate levels of energy
mining the benefits of efficiency-induced avoided emissions. demand reduction—equivalent to the continuation of current
The other major potential avoided emissions are greenhouse
policies—could lower total compliance costs,reduce rate-
gases(GHGs)—primarily carbon dioxide(CO2)—from fossil fuel
payer bills over the long term,and,in some cases,delay or
combustion. Energy efficiency is very important for reducing
avoid the need for equipment upgrades or new construction
GHGs because there are few options or"controls"for reducing of generating facilities and emissions controls.This energy
COz emissions from combustion once the COz is formed,and demand reduction is also likely to reduce emissions of air
because,unlike the pollutants mentioned above,the impact of GHG
pollutants on high electricity demand days when air quality
reductions is not location dependent.Therefore,energy efficiency
can be especially harmful.
can be the lowest-cost option for reducing GHG emissions.The Source:U.S.Environmental Protection Agency.(2012).The Roadmap for
importance of efficiency also becomes clear in light of the fact that Incorporating Energy Efficiency/Renewable Energy Policies and Programs into
approximately 60%of all human-induced (anthropogenic)GHG State and Tribal Implementation Plans.wwwepa.gov/airquality/eere.
emissions come from energy-related activities.41
6-1 www.seeaction.energy.gov December 2012

The federal government(including EPA and DOE)and many states 6.2 SPECIAL CONSIDERATIONS FOR
have recognized the value of incorporating end-use energy efficiency CALCULATING AVOIDED EMISSIONS FROM
into air regulatory programs.Several programs—including EPA's EFFICIENCY PROGRAMS
Acid Rain Program,EPA's NOx Budget Trading Program,and the This section describes some critical considerations for efficiency
Regional Greenhouse Gas Initiative(an effort of nine states from the program evaluators to consider when determining avoided emis-
Northeast and Mid-Atlantic regions)—have provided mechanisms for sions.One important consideration for both the avoided emissions
encouraging energy efficiency.43 The EPA has also provided guidance calculation approaches listed above is that the net energy savings
on incorporating energy efficiency into SIPs44 and has approved the calculated for the purposes of an energy resource program may be
inclusion of energy efficiency measures in individual SIPs.45 different from the net savings that need to be calculated to meet the
requirements of an avoided emissions program.The following are
For any type of energy efficiency program,the avoided air emissions three potential causes of the difference:
are determined by comparing the emissions occurring after the
• Different definitions of additionality
program is implemented to an estimate of what the emissions would
have been in the absence of the program(i.e.,emissions under a • Different definitions of boundary areas
baseline scenario).Conceptually,avoided emissions are calculated • Characteristics of emissions control mechanisms/regulations
using the net energy savings calculated for a program and one of two that may be in place.
different approaches:
1. Emission factor approach.This approach involves multiplying The first two items are discussed in Sections 6.2.1 and 6.2.2,respectively.
the program's net energy savings by emission factors(e.g., The"cap-and-trade"emissions control mechanism and its attributes
pounds of CO2 per MWh) representing the characteristics of with respect to energy efficiency are discussed in Section 6.2.3.
displaced emission sources to compute hourly,monthly,or Although it is not the only option to achieve widespread emission
annual avoided emission values(e.g.,tons of NOx or CO2 per year). reductions,it is addressed here because of its unique characteristics
There are several sources of emission factors,as well as approaches and its current application in the United States for controlling both
for calculating the factors.The emission factors approach can criteria pollutants and GHG emissions.
be used with any project type or energy resource,such as Following these sections is a brief overview,in Section 6.2.4,of the
boiler retrofits that save fossil fuels or lighting projects that possible objectives associated with calculating avoided emissions and
save electricity. Below is the basic equation for this approach: how they can affect decisions about which calculation approaches
avoided emissions=(net energy savings)x(emission factor) should be used and which specific issues should be addressed.
2. Scenario analysis approach.This approach involves calculating
a base case of source(e.g.,electricity generating units connected 6.2.1 Additionality
to a grid)emissions without the efficiency programs and Additionality is the term used in the emissions mitigation industry
comparing that with the emissions of those sources operating for addressing the key question of whether a project will produce
with the reduced energy consumption associated with the reductions in emissions that are additional to reductions that would
efficiency programs.This is done with a range of approaches, have occurred in the absence of the program activity.Note that
from using historical generation and load data or capacity additionality is typically defined as project-based,versus program-or
factor data with emission rates in spreadsheet calculations portfolio-based,and is directly related to the efficiency evaluation
to the use of sophisticated computer simulation approaches issue of defining proper baseline conditions and in some cases to
using"dispatch models."Scenario analysis is typically only used participant free ridership,as described in Chapter 5.As with baselines,
with large-scale,electricity-saving programs. Below is the basic additionality cannot be directly measured and must be inferred from
equation for this approach: available information.
avoided emissions=(base case emissions)—(reporting While the basic concept of additionality may be easy to under-
period emissions) stand,there is no common agreement on the procedures for
More information about each of these approaches and their applications defining whether individual projects or whole programs are truly
is provided later in this chapter. additional(i.e.,different than a baseline scenario).As such,there
is no technically correct level of stringency for additionality rules.
Evaluators may need to decide,based on their policy objectives,
December 2012 www.seeaction.energy.gov 6-2

what tests and level of scrutiny should be applied in additionality Direct and indirect emissions are two categories for consideration
testing.For example,program objectives that focus on formally when setting an emissions assessment boundary. Direct emissions
claiming avoided emissions benefits as part of a regulatory program are changes in emissions at the site(controlled by the project
may necessitate stringent additionality rules.On the other hand, sponsor or owner). For efficiency projects affecting onsite fuel use
programs that are primarily concerned with maximizing energy (e.g.,high-efficiency,fossil fuel water heaters or boilers),the avoided
efficiency and only need to approximately indicate avoided emissions emissions are direct. Indirect emissions are changes in emissions
may establish only moderately stringent rules. that occur at a source away from the project site(e.g.,of an electric
generating unit,or EGLI). Indirect emissions are the primary source
One area of common consideration for additionality is whether of avoided emissions for electrical efficiency programs.
an efficiency program is mandated by existing legislation(e.g.,an
energy efficiency resources standard),regulation(e.g.,a building When defining the assessment boundary,one must also consider
code),or policy(e.g.,a government agency's or private company's intended and unintended consequences,also called primary and
policy to reduce energy use by a certain percentage through efficiency secondary effects:
actions).That such legislation,regulation,or policy was influenced by . primary effect:the intended change in emissions caused by
anticipation of an entity being required to meet a future greenhouse a program. Efficiency programs generally have only one
gas reduction target or simply to improve air quality would argue for primary effect—energy savings at facilities that consume
such efforts as being additional. However,that such efforts will occur energy,translating into avoided emissions.
irrespective of additional support from an emissions program would
• Secondary effect:an unintended change in emissions caused
argue against such an additionality finding.This conflict is illustrative
by a program.Secondary effects are sometimes called "leak-
of the policy decisions to be made in deciding additionality.46
age"Leakage48 and interactive effects49 are similar concepts,
One approach for addressing the additionality conflict is to review although leakage is a more"global"issue,whereas interactive
what was assumed in terms of baseline energy use and efficiency effects tend to be considered within the facility where a project
activity when a greenhouse gas or air emissions target is set.Levels takes place.Two categories of secondary effects are one-time
of efficiency activity included in an air emissions baseline might very effects and upstream and downstream effects:
well not be considered additional once an air emissions target is — One-time effects:changes in emissions associated with
set based on such a baseline;that is,the level of emissions already the construction,installation,and establishment or the
assumes that level of efficiency activity.However,if the efficiency decommissioning and termination of the efficiency projects—
activity is beyond what is included in the baseline,then the program net of the same level of efficiency activity in the baseline
may be considered additional. scenario.
Upstream and downstream effects:recurring changes in
6.2.2 Assessment Boundary Issues: Primary and emissions associated with inputs to the project activities
Secondary Effects/Direct and Indirect Emissions (upstream)or products from the project activity(down-
The emissions assessment boundary is used to define and encom- stream) relative to baseline emissions. For example,one
pass all the energy uses and emission sources affected by activities potential upstream effect of possible concern (however
in a program.47 For avoided air emissions,the assessment boundary unlikely)for efficiency programs is that if efficiency programs
can be much larger than the boundary for calculating energy and displace energy sales and emissions in one area,the same
demand savings,including changes to emission rates and volumes amount of energy consumption and related emissions
beyond avoided emissions associated with lower energy use at the might be shifted elsewhere.A scenario for this would be if
efficiency project sites.This is particularly important for calculating production at the factory that underwent the efficiency
avoided emissions associated with electricity efficiency projects, actions is shifted to a factory outside the boundary area
when the affected facilities are connected to the grid. Because power or to one in the boundary with less-efficient energy use.
plants are interconnected through the electric grid system,to fully
understand electricity-related emissions requires an understanding Secondary effects,outside the facility where the efficiency project
of the operation of power plants throughout the system,which takes place,are typically minor relative to the primary effects of
usually involves regional considerations.See Section 6.4. energy efficiency programs—particularly when compared to baseline
secondary effects. For example,the manufacturing,maintenance,
6-3 www.seeaction.energy.gov December 2012

and installation of energy-efficient motors have no meaningfully The level of the cap is an important aspect of a cap-and-trade
different associated emissions than the emissions associated with program. In general,emissions may not exceed the cap,and they are
standard efficiency motors. In some cases,however,it is possible that also unlikely to be below the cap during any substantial period of
secondary effects could undermine the primary effect;therefore, time.The reason for this is that a unit that emits fewer allowances
the emissions assessment boundary should be defined and leakage than it has available may sell those allowances to another unit,which
possibilities considered,even if only documented that there are no will then use them to emit.54
significant,identifiable,secondary effects.One way to document such
secondary effects would be to conduct a generic study of the relative The fact that capped emissions tend to remain at the cap level is
upstream(and other)emissions associated with energy-efficient very relevant to the effect of energy efficiency,which reduces the
products versus conventional products and apply the results as a output of electricity generators or perhaps very large industrial boilers
"factor"to the calculation of avoided emissions(if such a factor were that could be subject to a cap.When emissions are not capped,
reliable and relevant). energy efficiency reduces emissions.As noted,this is not typically
true for emissions from sources subject to caps(e.g.,large boilers,
In summary,when evaluating the avoided reductions associated with power plants).This is because reductions in the output of electricity
efficiency programs,it is important to properly define the assessment generators do not alter the overall cap on emissions from electricity
boundary and to account for all primary effects(the intended savings), generators,and any reductions in emissions(and demand for allow-
as well as all direct emissions(at the project site)and indirect emissions ances)at a particular generator as a result of energy efficiency make
(at other sites).In addition,ideally,secondary effects should be at least extra allowances available for other entities to use.This means that
qualitatively investigated. freed-up allowances can be sold in the market and used elsewhere
or banked for use in a later year,such that total emissions will remain
6.2.3 Special Issues for Capped Pollutants roughly equal to the cap level.
under Cap-and-Trade Programs
There are several regulatory mechanisms for controlling pollutants The goal of the cap-and-trade program is typically not to go below the
and greenhouse gas emissions,and cap-and-trade is one of them. cap but to achieve the cap at the lowest possible cost to society,so
Under a cap-and-trade program,an overall emission tonnage cap is energy efficiency contributes to the primary goal of the cap-and-trade
set for an affected sector or set of facilities.Allowances are created to program by helping to achieve the emissions target while minimizing
represent the emission of each unit(e.g.,1 ton)of pollution under the compliance costs. In addition,efficiency programs may reduce emis-
allowable cap.The primary compliance requirement is that each plant sions from non-capped emission sources and non-capped pollutants
must hold allowances equal to its actual emissions at the end of each and directly claim avoided emissions if properly calculated.
compliance period.However,there is no fixed emissions cap or limit
on an individual facility,and each facility's emissions are not limited to There are,however,mechanisms by which efficiency programs under
the allowances that it initially receives or buys at auction(depending a cap-and-trade system can claim avoided emissions for capped
on how allowances are allocated). It may purchase additional allow- pollutants.The primary mechanism is that allowances are retired,or
ances from another facility or sell allowances if it has a surplus. removed from the market.In states that have established an energy
efficiency set-aside program(a pool of allowances from within the
There are several examples of U.S.cap-and-trade programs: cap that is set aside and made available for energy efficiency project
• The Title IV acid rain sulfur dioxide(SO2)trading program sets and program implementers)as part of their cap-and-trade program
a cap on annual SO2 emissions for U.S. power plants.so design,efficiency program implementers may acquire allowances to
•
The Clean Air Interstate Rule(CAIR)includes a cap-and-trade retire by submitting a claim for allowances to the set-aside adminis-
mechanism for power plants in the eastern United States to trator.Alternatively,efficiency programs may acquire allowances to
achieve reductions of emissions that cross state lines and retire by purchasing them from other market participants.
contribute to ground-level ozone and fine particle pollution For example,some states have created special set-aside allocations
in other states.51 of allowances in their NOx trading programs for energy efficiency
• CO2 emissions from power plants are capped in the nine states projects.55 Qualified project sponsors that obtain these allowances
of the Northeastern Regional Greenhouse Gas Initiative52; in can choose to retire them to make emissions-reduction claims and
2011,California enacted a multi-sector cap-and-trade program avoid the expense of an allowance purchase that would otherwise be
to limit GHG emissions.53 necessary to make such claims. However,sponsors may also sell the
December 2012 www.seeaction.energy.gov 6-4

allowances to finance the efficiency project,in which case they may Calculating avoided emissions for regulatory purposes or a
not claim the reduction. Under CAIR,states have an opportunity to primary program objective.Rigorous analyses are appropriate
develop energy efficiency(or renewable energy)set-aside programs. when avoided emissions are a primary goal of an efficiency
program—typically,when the efficiency program is part of a
Lastly,it should be noted that while efficiency generally does not regulatory scheme or is intended to generate creditable emission
reduce aggregate emissions for capped pollutants,efficiency may reductions or offsets with a significant monetary value or to
affect the temporal or geographic distribution of emissions in ways comply with a regulatory mandate. In these situations,docu-
that bring environmental benefits,as well as possibly reduce the mentation should be provided(either on a project-by-project
on-site emissions footprint of the entity undertaking the efficiency basis or,preferably,on a program or policy level)56 that the
actions. For example,efficiency programs that target cooling loads energy savings(probably net energy savings)and avoided emis-
may reduce electric generating unit NOx emissions on hot summer sions are additional or surplus as defined by the air regulator.
days with high ozone concentrations in ways that reduce pollution A boundary definition is also desirable to document that there
levels,even though aggregate NOx emissions for the ozone season is no anticipated emissions"leakage."That boundary definition
would be unchanged.For more information on the air-pollution properly defines the locations of the avoided emission sources—
impacts of energy efficiency under capped systems,visit from the efficiency project sites to within a multi-state electric
www.epa.gov/airquality/eere.html. grid. In the case of regulatory mandated air emissions control
6.2.4 Avoided Emissions Calculations for programs,the methods for calculating avoided emissions may
Different Objectives be defined. In terms of the actual emissions-reduction calcula-
Avoided emissions calculations have a wide range of specific applica-
tions,one of the more complex approaches described below
tions,such as voluntary and mandatory GHG offset programs and will most likely be appropriate.
NOx cap-and-trade programs with energy efficiency allowance set- 6.3 DIRECT ONSITE AVOIDED EMISSIONS
asides.These programs have varying requirements for documenting CALCULATIONS
legitimate avoided emissions.Those interested in creating tradable
offsets,allowances,or other program-specific credits should consult Direct onsite avoided emissions can result when efficiency programs
the regulations of the specific program they are interested in with save self-generated electricity that would have been produced at
respect to additionality, boundary area definitions,and other issues a project site or,more typically,when efficiency reduces the need
specific to the program. for onsite heat or mechanical energy,reducing onsite combustion
of natural gas,fuel oil,or other fuels.Identifying the appropriate
However,the following are some rule-of-thumb recommendations, emission factor is fairly straightforward for onsite emissions,such
organized by objective,for calculating the avoided emissions: as those from residential or commercial combustion equipment,
Calculating avoided emissions primarily for informational industrial processes,or onsite distributed generation.The emission
• purposes.When the primary goal of an efficiency program is factors are commonly calculated in one of two ways:
to save energy or demand,the avoided emissions are often Default emission factors. Default emission factors are based
reported only subjectively or with minimal analysis,to indicate on the fuel and emission source being avoided.This is the most
a co-benefit of the program. In this situation,the expectations common approach and a wide variety of resources provide
for the certainty of the avoided emission values are not high, emission factors per-unit of fuel consumption,including
and the avoided emission estimates are not used in a regula- manufacturer's equipment performance data,state-certified
tory or market scheme where a monetary value is ascribed to performance data,emission permit data,and generic emission
the avoided emissions.Thus,one of the simpler approaches data compiled by regulators or industry groups.
described below is probably appropriate. It is typical that(1) A standard resource is the EPA's AP-42,Compilation of Air
additionality is simply assumed, (2)emissions boundary area Pollutant Emission Factors.57 It is the primary compilation of the
issues are not necessarily rigorously addressed,and(3)the EPA's emission factor information and contains emission factors
energy savings are simply those reported for the program, and process information for more than 200 air pollution source
whether net or gross.These savings are then multiplied by categories.A standard resource for GHG emissions is the 2006
appropriate, preferably time-dependent,emission factors to IPCC Guidelines for National Greenhouse Gas Inventories.58
calculate avoided emissions.
6-5 www.seeaction.energy.gov December 2012

• Source testing.Source testing can determine the emission Finally,annual avoided emissions is calculated:
factors for a specific device(e.g.,large-scale industrial boil- - 125,000 therms/year x 0.005 metric tons CO2/therm-
ers). Protocols for testing are available,but given the time and 625 metric tons COz equivalent/year.
cost of such testing,this approach is usually only taken when
required by environmental regulation.This may change if the This example is provided,in part,to demonstrate that even some-
value of avoided emissions makes source testing costeffective thing as simple as an emission factor calculation can be complex
as a part of a certification process. and involve more than just reading an emission factor off a chart.
It is important to understand the assumptions and applicability of
The program evaluator must select onsite emission factors that the factor,such as in the above case for calculating COz emissions
provide sufficient accuracy to meet the goals of the evaluation.This (equivalent)from a carbon emissions value.Therefore,it is always
requires selecting different emission factors for different time periods, suggested that people familiar with the use of such emission factors
places,and technologies. In addition,emission factors based on be consulted when doing this type of analysis.
historical emission rates may need to be adjusted to account for new,
more stringent regulations.Accounting for changing environmental 6.4 EMISSION FACTORS FOR GRID-CONNECTED
regulation is an important consideration in calculating emissions. ELECTRIC GENERATING UNITS
The following is an example of an avoided COz emissions calculation As with the direct onsite emissions cases,emission reductions
for a project that reduces natural gas consumption from a large from reduced electricity consumption occur because less fuel is
industrial boiler. combusted. However,calculating avoided electrical grid emission
reductions is more complex because the fuel combustion in question
• First,avoided natural gas use is calculated, in units of therms would most likely have occurred at many different existing or
per year(1 therm=100,000 Btu): future electric generating units(ECUs)connected to the grid.59Thus,
- Displaced steam use due to efficiency project=10,000 emissions from displaced electricity usually depend on the dynamic
million Btu(MMBtu)/year interaction of the electrical grid,emission characteristics
- Steam boiler higher heating value(HHV)efficiency=80% of grid-connected power plants,electrical loads,market factors,
- Displaced natural gas use=10,000 MMBtu/year-0.80-_ fuel and electricity economics,and a variety of regional and environ-
12,500 MMbtu/year=125,000 therms/year. mental regulatory factors—all of which can change over time.
• Next,an emission factor is calculated: When using electricity savings values for calculating avoided emis-
- The average carbon coefficient of natural gas is 14.47 sions,a critical consideration is to convert the electricity savings
at the site when the efficiency measures are implemented to the
kilograms(kg)of carbon per MMBtu.The fraction oxidized
to COz is 100%(from the Intergovernmental Panel on
electricity savings at the EGU.This means taking into account trans-
Climate Change[IPCC]). mission and distribution(T&D)losses between the end use(e.g.,
home,office,factory)and the generator.The difference between
- COz emissions per therm are determined by multiplying site savings and savings at the generator(sometimes called source
the following: natural gas heat content;the carbon(C) or busbar savings)varies with a wide range of factors,with typical
coefficient;the fraction oxidized;and the ratio of the values of 5%-20%.
molecular weight ratio of carbon dioxide to carbon(44/12).
When using this equivalency,keep in mind that it represents This section of Chapter 6 first provides some basic information about
the COz equivalency for natural gas burned as a fuel, not the electric grid and the relationship of efficiency-displaced existing
natural gas released to the atmosphere. Direct methane (and perhaps future)electricity generation and avoided emissions.
emissions released to the atmosphere(without burning)are This is followed by descriptions of several approaches,from simple
about 21 times more powerful than COz in terms of their to complex,for calculating avoided emissions.
warming effect on the atmosphere.
- MMBtu/1 therm x 14.47 kg C/MMBtu x 44 grams(g) 6.4.1 The Electricity Generation Mix
CO2/12 g C x 1 metric ton/1,000 kg=0.005 metric tons The electric grid is composed of a T&D system,often covering multiple
CO2-equivalent/therm. states,connecting a mix of generating plants with different emissions
characteristics,which operate at different times to meet electricity
December 2012 www.seeaction.energy.gov 6-6

demand.The mix of plants operating varies by region and overtime FIGURE 6.1: Example electricity load duration curve
within regions—both as the demand changes from one hour to the
next and as old plants are retired and new plants are built.A common
18,000 --------------------------------------------
way of looking at this varying generation mix is a load duration curve. 16,000 ------------------------------------
The load duration curve shows the electricity demand in MW for a Peaking Turbines
14,000 -------------------------------------------
regionfor each of the 8,760 hours in the year.The hourly demand
values are sorted from highest to lowest.Figure 6.1 shows an example 12,000 --- ----------------------------------------
from a typical east coast electric utility.
Cycling Coal,Oil,Gas
The figure shows that the highest hourly electric demand was16,000 85000 ------- --------------- -------------
MW and the lowest was 5,500 MW.It also shows that the peaking Gas Combined Cycle
6,000 --------- ----
turbines and reciprocating engines operated for only about 200 Base Load Coal
hours per year(in this case,during very hot hours of the summer), 45000 --------------------------------------------
while the baseload coal and nuclear plants operated throughout 25000 --------------------------------------------
the year.The total area under the curve is the generation needed to Nuclear
meet load plus line losses(in this case,about 80 million MWh).The —
varying electric load is met with a large number of different types 0 15000 2,000 3,000 4,000 5,000 6,000 7,000 85000
and sizes of generating units. HOURS
Figure 6.1 also indicates a typical mix of generating technologies.
The generating units are dispatched based on a number of factors, The load duration curve in Figure 6.1 depicts an existing generation
the most important usually being the unit's variable cost—the mix. However,efficiency could also prevent the need for future
cost of fuel along with operation and maintenance directly related power plant construction. For most energy efficiency program activity
to production and perhaps regulation-based loading orders that in the United States,it is safe to assume that only existing generator
prioritize certain resource types,such as renewables. Baseload units emissions are avoided in the short term of one to five years.However,
are operated as much as possible,unless there is an environmental if the analysis is estimating impacts over a longer period of time and/
regulation limitation,because they are the least expensive.On the or the scale of the programs being evaluated is large enough,then
other hand, peaking and intermediate(cycling)units are used only new units could be considered as well.
when needed because of their higher costs.The type of units—such
as baseload or peaking—that are the most"polluting"can vary from The emission factor from a generating unit that would not be run due
one region to another. to energy efficiency is called the operating margin(OM).The emis-
sion factor from a generating unit that would not be built is called
Compared to the base case,energy efficiency displaces a certain the build margin(BM).In general terms,avoided emissions can be
amount of generation during each hour that it operates. Efficiency estimated by determining the extent to which an efficiency program
essentially takes a"slice"off the top of the load curve for the hours or portfolio affects the BM and OM and either(1)determining
that it occurs,displacing the last unit of generation in each of these appropriate emission factors for the BM and OM using the emission
hours.The displaced emissions can be estimated by multiplying the factor approach,or(2)accounting for new and existing generating
displaced generation by the specific emission rate of that unit or by units when using the scenario approach.This is discussed further in
preparing scenario analyses. the following subsections.
Depending on the hour of the day or year and the geographical The general formula for calculating emission rates for determining
location of the avoided electricity use,the displaced unit could be avoided emissions is:
a cycling coal,oil,or steam unit;a combined-cycle unit;a central ER=(w)x(BM)+(1—w)x(OM)
station peaking turbine;or a reciprocating engine unit—or even a
zero-emissions unit.The first challenge in calculating the avoided Where:
emissions for electricity generation is defining the mix of technologies ER is the average emission rate(e.g.,tons of
displaced by the efficiency programs for the specific program location CO2-equivalent/MWh)
and during specific times of the year.
6-7 www.seeaction.energy.gov December 2012

• w is the ratio(between 0 and 1)assigned to the build margin If the BM is included in the analyses,it must be explicitly specified,
• BM is the build margin emission factor(e.g.,tons of including the basis for its calculation and justification for its use as the
CO2-equivalent/MWh) most likely scenario. In recent years,estimates for BM emission rates
• OM is the operating margin emission factor(e.g.,tons of have been based on advanced-technology coal plants or gas-fired,
CO2-equivalent/MWh). combined-cycle power plants,as most new thermal plants adopt this
technology. However,with new technologies being developed and
Time is explicit in the above equation.That is,the emission reduction renewable portfolio standards becoming more prevalent,changes in
can vary from year to year(or in theory from hour to hour)as the market conditions should be tracked and accounted for when using a
variables w,BM,and OM change over time. In this equation,w BM emission factor.
indicates where the generation produced(or reduced)by the project 6.4.2 Using Avoided Emission Factors and Scenario
activity would have come from in the baseline scenario.A ratio(w) Analyses for Electric Grid Analyses
of 1 means that all generation produced or saved by the project The methods for determining avoided emissions values for displaced
activity would have come from an alternative type of new capacity
generation range from fairly straightforward to highly complex.They
built in place of the project activity(the BM).A ratio between 0 and
include both spreadsheet-based calculations and dynamic modeling
1 means that some of the generation would have come from new
approaches with varying degrees of transparency,rigor,and cost.
capacity(BM)and the remainder from existing capacity(the OM).A Evaluators can decide which method best meets their needs,given
ratio of 0 means that all of the generation would have been provided
evaluation objectives,available resources,data quality requirements,
by existing power plants,and no new capacity would have been built
and evaluation framework requirements.Designers of programs or
in place of the project activity.
regulations that use these estimates may also wish to specify one
One approach to determining OM and BM can be found in the or more methods at the outset,and a process for periodic review of
World Resources Institute/World Business Council for Sustainable those methods.
Development(WRI/WBCSD)Protocol Guidelines for Quantifying
The emission rates of the electric grid will vary over time.Thus,the
GHG Reductions from Grid-Connected Electricity Projects.60 In this
emissions analyses are typically conducted annually for each year
WRI/WBCSD approach,there are three options for selecting the
BM emission factor:61 of the evaluation reporting period for electricity-saving programs.
Emissions rates can also vary hour by hour as the mix of electricity
• Option 1.Use a project-specific analysis to identify the type plants operating changes to meet changing loads.The decision to
of capacity displaced.Under this option,the BM emission use an annual average analysis,an hourly analysis,or some time
factor is representative of a single type of power plant.This type of period of analysis in between is up to the evaluator to decide,based
power plant will be either(1)the baseline candidate(i.e.,baseline on evaluation objectives and available resources,as well as evalua-
power plant)with the lowest barriers or greatest tion framework requirements.
net benefits,or(2)the most conservative,lowest-emitting baseline
candidate. The following are descriptions of four emissions quantification
• Option 2.Use a conservative"proxy plant"to estimate BM approaches.These four approaches are listed in order of increasing
emissions. Under this option,the BM emission factor is determined complexity of assumptions and sophistication of analysis:
by the least-emitting type of capacity that might reasonably be 1. Regional non-baseload emission rates(using ERAVs
built.In some cases,this baseline candidate could have an emis- eGRID database)
sion rate of zero(e.g.,renewables).Another way to determine 2. Regional marginal baseload emission rates(using capacity
a proxy is to look at the plants that have recently been built and factors or equivalent)
connected to the grid. 3. Regional historical hourly emission rates
• Option 3.Develop a performance standard to estimate the BM 4. Energy scenario modeling.
emission factor. Under this option,the BM emission factor will
reflect a blended emission rate of viable new capacity options. Table 6.1 provides some guidance on selection of an approach for a
given set of objectives and metrics.
December 2012 www.seeaction.energy.gov 6-8

TABLE 6.1: Choosing an Avoided Grid Electricity Emissions Quantification Approach
APPROACHEMISSIONS ENERGY DATA EMISSIONS
ANALYTICAL QUESTIONS QUANTIFICATION NEEDS OUTPUTS
What is the relative magnitude of emission Load emission rates Annual or seasonal Regional
reductions of the energy efficiency portfolio? (using EPA's eGRID energy impacts(MWh) non-baseload
database) avoided emissions
Which EGUs in my region are on the margin, Regional marginal Annual or seasonal Regional marginal
and how much seasonal or annual emissions will baseload emission energy impacts(MWh) unit avoided
be avoided? rates(using capacity emissions
factors or equivalent)
How can I quantify hourly emission reductions? Regional historical Hourly energy impacts Regional hourly
How much are emissions reduced during peak hourly emission rates (MW and/or MWh) avoided emissions
electricity demand?
How will EGU emissions change in future years? Energy scenario Hourly,seasonal,or Regional avoided
How can I compare baseline and forecast emissions modeling annual energy impacts, hourly,seasonal,
that can result from efficiency portfolios? which depend on the or annual emissions
model used(MW and/ based on dispatch
How can I estimate avoided emissions in a cap-and- or MWh) order
trade program?
6.4.2.1 Regional Non-Baseload Emission Rates Approach Figure 6.2 shows the eGRID subregions,which are identified and
(Using EPA's eGRID Database) defined by EPA,using the North American Reliability Corporation
This approach entails a simple calculation—multiply the amount of (NERC)regions and power control areas as a guide.An eGRID subregion
generation or electricity sales displaced by the efficiency action by is often,but not always,equivalent to an Integrated Planning Model
the"non-baseload"emission rate indicated for a specific pollutant (IPM)63 subregion.The 26 eGRID subregions in eGRID2010 are
in a region.The non-baseload emission rate represents an average subsets of the NERC regions,as shown in Figure 6.2.
emission rate for the EGUs that are likely to be displaced by end-use
The eGRID subregion non-baseload output emission rates are used
efficiency actions.
to estimate emission reductions of end-use efficiency programs,
A standard source of emission rates for the United States is the EPA's portfolios,or policies that reduce consumption of grid-supplied
eGRID database.The eGRID database includes operational data such electricity. Non-baseload output emission rates are associated with
as total annual emissions and emission rates(for GHG, NCx,S02,and emissions from plants that combust fuel and are the EGUs most
mercury),generation,resource mix,capacity factors,and heat input. likely to back down when energy efficiency and renewable energy,
The eGRID emissions are associated with the generation of electric- policies,and programs are implemented.These emissions data are
ity,and thus the values do not account for T&D losses,imports and derived from plant-level data and are aggregated up to the eGRID
exports among eGRID subregions(or any other geographic area), subregion level.
transmission constraints within any geographic area,or lifecycle
emissions at EGUs(e.g.,emissions from the extraction,processing,
and transportation of fuels).
6-9 www.seeaction.energy.gov December 2012

FIGURE 6.2: U.S. EPA eGrid subregions
MROE
NEWE
MRO NYUP
RFCM
NWPP
NYU
RFCE NYCW
RFCW
RMPA CAMX SRMW
SPNO
SRTV SRVC
SPSO
AZNM
SRMV SRSO
AKMS
ERCT
H OA
♦ �� FRCC
AKGD �HIMS
Source:U.S.Environmental Protection Agency.The Emissions&Generation Resource Integrated Database(eGRID).www.epa.gov/egrid/
The main advantages of the eGRID subregion non-baseload emission The bottom line is that this is an easy approach to apply, but the
rates approach are that it is a straightforward,simple calculation and tradeoff can be relatively high uncertainty in the estimate of the
that it can be used for communicating to the public that emission emissions-related impacts. In summary,the advantages of this
reductions can result from the implementation of efficiency programs. approach are as follows:
A possible shortcoming of this approach is that energy efficiency . It provides an easy"back of the envelope"calculation
savings tend to vary over time,such as savings from an office lighting
• Non-baseload output emission rates provide a basic
retrofit that only occurs during the workday or an exterior lighting
understanding of how much EGU emissions could likely
retrofit with savings only at night;thus,using an annual average emis
be avoided or displaced.
sion factor that lumps daytime,nighttime,weekday,and weekend
values together may skew the actual emissions benefits calculation. And,in summary,this approach also has some limitations:
A system-average emission rate may also be purely historical,and • Future-looking EGU representation is missing(i.e.,there is no
thus fail to account for changing emissions regulations and new "build margin"analysis)
plant additions. Historical system averages will tend to overestimate • Some EGUs in the base year may have already shut down or
emissions impacts if emissions limits become more stringent over will do so in future years
time.Alternatively,a system-average emission rate could be estimated • The eGRID approach uses averages and does not show where
for a hypothetical future system,based on assumptions about emissions or which EGUs will be displaced
from new plants and future regulatory effects on existing plants. 0 Information is generally on a 3-year time lag.
December 2012 www.seeaction.energy.gov 6-10

• The eGRID approach only accounts for generation within a seasonal weekday and weekend and nighttime and daytime values
specific area and does not include information about imports/ (i.e.,six emission factors)to match up the net efficiency savings for
exports of electricity(except for state-level net imports) the equivalent time period will significantly improve estimates over
• The approach assumes that efficiency activities will affect all the other emission factor methods previously described above.
non-baseload plants proportionally to each plant's non-baseload
generation. One of these marginal baseline approaches is to use EGU capacity
factor data.An EGU's capacity factor is the ratio,in a given period of
6.4.2.2 Regional Marginal Baseload Emission Rates time,of the actual electricity produced by a generating unit to the
Approaches (Using Capacity Factors or Equivalent electricity that could have been produced at continuous full-power
Approaches) operation.The capacity factor of an EGU can be used as a proxy for
These approaches have been developed to provide a reasonably how likely the EGU is to be displaced by end-use efficiency actions.
accurate estimate of displaced emissions at a lower cost than The approach helps users understand the relative dispatch order of
analyses using the next two approaches,historical hourly data or
the EGUs within a state or group of states.After doing this analysis,
modeling,but with more accuracy than the simple,non-baseload one has an estimate of which EGUs are on the margin(essentially
approach discussed above.Using regional marginal baseload emission
the load duration curve)and how much emissions could be displaced
rates typically involves using spreadsheets and compiling publicly from each EGU on a seasonal or annual basis.
available data to approximate which marginal generating units For example,this approach assumes that EGUs with low capacity
will be supplying power at the time that efficiency resources are factors(e.g.,operating at equal to or less than 20%of capacity)are
reducing consumption. most likely to be displaced by the efficiency portfolio,and EGUs with
The two major steps in a spreadsheet-based analysis are to(1)determine high capacity factors(e.g.,operating at equal to or greater than 80%
the relevant set of generating units(i.e.,account for the location of of maximum capacity)would not be displaced by the efficiency activ-
the efficiency program's projects as well as transfers between the
ity.When available,seasonal capacity factors should be used instead
geographic region of interest and other power areas),and(2)estimate
of annual capacity factors,which ignore seasonal weather variations.
the displaced emissions from those units. For example,many combustion turbines only operate during summer
daytime hours in a typical year. Using an annual capacity factor
As discussed above,generating units are typically dispatched in a would incorrectly allocate displaced emissions to these units during
predictable order,based on cost and other operational characteristics. seasons when they are not operating.
This means it is possible,in principle,to predict which unit types will
be"on the margin"at a given load level,and thereby predict the In summary,the advantages of these approaches are as follows:
marginal emission rates. Data on regional power plants may be used • Emissions can be assigned to each EGU
to develop supply curves representing different seasons and times of • The calculation is relatively easy if the analysis infrastructure is
day.These curves are then used to match regional electricity loads to set up
characteristic emission rates.Although this method can use readily • It is a simple way to get a relative sense of the marginal unit in
available public data,it is based on a simplified view of the dispatch the area of analysis.
process that does not account for transmission congestion.
These approaches also have Imitations:
As with the system-average approach,this method does not provide Annual capacity factors assume that the EGUs operate the
a way to determine how large a geographic region should be same throughout the year
considered or how interregional transfer is estimated. However,this
approach improves upon the system-average approach with respect • Emissions estimates are approximate,based on annual or
to identification of marginal generators. In either case,the analysis seasonal capacity factors,and they do not account for
must include the effect of changing environmental regulation,as maintenance or outages
discussed above. • Imported and exported power is not considered
• They assume that EGU generation characteristics are the
A significant advantage of using time-varying emission rates is same in the base year and future years
that they can match up to the time-varying savings from efficiency
programs.Even if an hour-by-hour load shape is not used,having
6-11 www.seeaction.energy.gov December 2012

• They assume that all energy savings or generation affect Other examples of data are available as well:
all peaking units first,which is not always true(e.g.,street • The Mid-Atlantic Regional Air Management Association
lighting programs). (MARAMA)completed an hourly emissions analysis of the
As an example of this approach,since 1993, ISO New England Inc. states in the Northeast and Mid-Atlantic region.65
(ISO-NE)has annually analyzed the marginal emission rates of the • The Metropolitan Washington Council of Governments used
New England electric generation system.This is motivated by the a time-matched marginal emissions approach that matches
need to determine the emission reductions that demand-side man- certain efficiency(and renewable)technologies or measures
agement(DSM)programs have had upon New England's aggregate with historical hourly emissions information from the EPA
NOx,SOz,and COz generating unit air emissions.The use of these hourly database.This emissions tool can be used for the
emission rates was subsequently broadened to include the benefits Virginia/Maryland/Washington, D.C.area.66
of renewable resource projects in the region.The 2010 report is To use this approach,one needs to identify whether the efficiency
available at http://iso-ne.com/genrtion_resres/reports/emission/
affects peak hours and/or baseload energy use.The evaluator should
fina I_2010_em issions_report_v2.pdf.
add the programs together in a"bottom up"approach to obtain an
6.4.2.3 Regional Historical Hourly Emission Rates Approach aggregate level of energy savings and generation on an hourly basis,
and then apply their impacts to the predicted displaced EGUs.The
This approach requires technical manipulation of historical generation, steps associated with this approach are as follows:
load,and emission rates to determine EGU dispatch order and marginal
Collect EGU hourly emissions data from EPA or other resources.
emissions rates.By applying this approach,one can determine
where each EGU fits within the dispatch order for each hour,day, 2. Determine EGU dispatch order(and thus hourly emissions)and
and month of a historical year—and thus determine which EGU's prepare hourly"bins"of emission rates.
emissions are avoided by less energy demand,due to efficiency,in a 3. Apply hourly end-use energy efficiency savings,with a T&D
given hour.This approach does not account for electricity imports or factor,to the"bins"of emission rates to quantify hourly
exports into or out of the grid being studied,nor does it specifically avoided emissions as a result of the efficiency activities.
address transmission constraints.
The advantages of this approach are as follows:
With this approach,one can understand (for every hour or segment . Reported data are easy to find on EPA's website on an hourly,
of hours of a historical year)which EGUs are baseload(operating daily,and quarterly basis
all hours of the day),which EGUs are load-following(EGUs that
• Emission rates from any group of hours can be derived from the
ramp up or down depending upon demand),and which EGUs are
hourly data.
peaking units(EGUs that only operate at high demand periods).This
approach is most appropriate to answer questions such as:"How This approach also has limitations:
much emissions are reduced in blocks of hours,or during periods of
• Setting up an hourly emissions database can be resource
peak electricity demand?"or"How much emissions are reduced for
intensive if the infrastructure is not established
demand-response policies?"
• Representation of future EGU emissions is missing
The EPA has information that can be used for this approach.It col- . Only EGUs subject to EPA's national reporting requirements
lects generation,emissions,and heat input data in hourly intervals are represented in its hourly database
from continuous emissions monitoring systems(CEMS)for all Energy import and export exchanges and transmission
large EGUs subject to EPA trading programs.For example,the EPA constraints are not captured.
implements the emissions cap-and-trade program for the Acid Rain
Control Program,the NOx Budget Trading Program,and the Clean 6.4.2.4 Energy Scenario Modeling Approach
Air Interstate Rule.These programs require an hourly accounting
At the other end of the complexity spectrum from calculating
of emissions from each affected unit.Affected units are sources
simple average emission factors are the energy scenario modeling
participating in CEMS that provide hourly emissions data unless the
approaches that use dynamic simulation models of the grid.These
EGU qualifies to use one of the alternative monitoring methodolo-
are generically called dispatch models.Varieties or different names
gies specified in EPA rules.64
for dispatch models include capacity expansion models,production
December 2012 www.seeaction.energy.gov 6-12

cost models,and system planning models.Dispatch models fore- an anticipated future system based on detailed assumptions about
cast which EGUs will operate at any given time based on inputs additions,retirements,and major grid changes(e.g.,capacity
and assumptions in the model,and their algorithms simulate the expansion or system planning models). However,dispatch models
complex interactions of the grid with consideration of factors such do not model the competition among different generating technolo-
as transmission constraints,import/export dynamics,fuel prices,air gies to provide new generation. In general,the model produces a
pollution control equipment,and a wide range of energy policies and deterministic, least-cost-system dispatch based on a highly detailed
environmental regulations. Dispatch models specifically replicate representation of generating units—including some representation
least-cost system dispatch,with the lowest-cost resources dispatched of transmission constraints,forced outages,and energy transfers
first and the highest-cost last.All of these models can capture a high among different regions—in the geographic area of interest.
level of detail on the specific ECUs displaced by energy efficiency
projects or programs. Dispatch models using only data for existing units are generally good
to use for projecting avoided emissions 1-5 years into the future,
The models are used to generate scenarios of the electric grid's especially when the future EGU fleet is not changing substantially.
operation and emissions. If the power system is altered through load Capacity expansion models forecasting future generation and retire-
reduction or the introduction of an efficiency program,the model ments,as well as the dynamic fluctuation within the electric grid,are
calculates how this would affect dispatch and then calculates the generally useful for analysis 5-30 years into the future.This approach
resulting emissions and prices.The basis for this scenario approach is most appropriate to use when a portfolio is expected to be large
is that a dispatch model is run with and without the efficiency enough to substantively change electric system operations.
actions,and the resulting difference in emissions is calculated.The
models can also be used to provide hourly,monthly,or annual The advantages of this approach are as follows:
emission factors. • It is the most sophisticated way to capture how the electrical
grid will react to implementation of efficiency actions
Dispatch modeling can be the most precise means of quantifying • Electricity transfers are well represented
avoided emissions(assuming good input assumptions and qualified
modelers)because it can model effects of load reductions that are • It can provide very detailed estimations about specific plant
substantial enough to change dispatch(as well as future changes and plant-type effects
such as new generating units or new transmission corridors)on an • It can provide highly detailed,geographically specific hourly
hourly basis,taking into account changes throughout the intercon- avoided emissions data at the EGU level.
nected grid.As such,it is a preferred approach where feasible. • Future EGU generation and retirements can be represented
On the downside,dispatch modeling typically involves the use of • It can be relatively inexpensive to use for an efficiency portfolio
proprietary,commercial programs; requires extensive underlying analysis if the models are already developed,and appropriate
data;and can be labor intensive and difficult for non-experts to
data populated for other purposes,such as supply-side
evaluate.These models can also be expensive,although the costs
generation analyses.
have been reduced over recent years and—particularly if the This approach also has limitations:
results can be applied to a large program or several programs—
the improved estimate can be well worth the incremental cost. The models are only as good as the assumptions used
Accordingly,they are probably most appropriate for portfolios that • Hourly emissions data are not always available as the
seek to achieve significant quantities of electrical energy efficiency or required input
long-term effects.For large statewide programs,the modeling costs • Energy models are proprietary,require significant resources to
may be relatively small compared to the program and evaluation run compared to other approaches,and can be data intensive
costs;the California Public Utilities Commission,for example,used . Input assumptions can be difficult to discern due to the
dispatch modeling to determine the avoided greenhouse gases from proprietary nature of the models and the amount and
various efficiency portfolios.67 complexity of assumptions and data used
An hourly dispatch model simulates hourly power dispatch to explic- • Expertise in energy modeling is normally recommended.
itly estimate emissions from each unit in a system.That system can
represent the current grid and generating units,or it can represent
6-13 www.seeaction.energy.gov December 2012

6.5 SELECTING AN APPROACH FOR For criteria pollutants,the following document from the EPA provides
CALCULATING AVOIDED EMISSIONS guidance on calculating avoided emissions for both energy efficiency
The choice of evaluation approach is tied to the objectives of the and renewables programs:The Roadmap for Incorporating Energy
program being evaluated,the scale of the program,the evaluation Efficiency/Renewable Energy Policies and Programs into State and
budget and resources,and the specific emissions the program is Tribal Implementation Plans,available at www.epa.gov/airquality/
avoiding.For direct onsite fuel savings and the resulting avoided emis- eere(see Appendix I of the roadmap).Additional EPA references are
sions,per common practice,standard emission factors can be used. provided in Appendix C.2.3.
For electricity savings programs,system average emission values can
be used,but they should be avoided except in the simplest estimates.
There are also medium-effort approaches(using capacity factors
or historical emissions rates)that can fairly accurately quantify the
effects of electricity energy efficiency programs.However,the most
sophisticated approaches involve dispatch modeling and the result-
ing detailed calculation of hourly emissions.While the costs and
complexity of these models has limited their use in the past,this is
beginning to change. Dispatch models are potentially cost-effective
evaluation tools that should be considered for evaluations of large-
scale programs.
6.6 ADDITIONAL RESOURCES ON AVOIDED
EMISSIONS CALCULATIONS
The following documents provide some guidance with respect to
greenhouse gas programs.Each is a product of the World Business
Council for Sustainable Development(WBCSD)and/or the World
Resources Institute(WRI)and is available at www.wri.org/climate.
• Guidelines for Quantifying GHG Reductions from Grid-
Connected Electricity Projects,www.ghgprotocol.org/files/ghgp/
electricity_final.pdf, published in August 2007
• GHG Protocol Corporate Accounting and Reporting Standard
(Corporate Standard),www.ghgprotocol.org/files/ghgp/public/
ghg-protocol-revised.pdf,revised edition,published in
March 2004
• GHG Protocol for Project Accounting(Project Protocol),
www.ghgprotocol.org/files/ghgp/ghg_project_protocol.pdf,
published in December 2005.
Examples of energy efficiency projects implemented for their
greenhouse gas emissions benefits can be found at the Climate Trust
website:www.climatetrust.org.
December 2012 www.seeaction.energy.gov 6-14

CALCULATIONOF • • •NSIN'S FOCUS ON •
Evaluators for Wisconsin's Focus on Energy(Focus)public benefits To identify marginal plants,the evaluators calculated the average
energy efficiency program estimated emission factors for the plants length of time,in hours,that a generating unit remains on once it
serving Wisconsin.They used these data to estimate environmental has been brought online. Peaking units,which are brought on for
impacts,in the form of displaced power plant emissions,associ- only a short time,have a short average time on;baseload plants
ated with Focus energy savings.The evaluation team developed that remain on for hundreds of hours or more have a long average
a model to estimate the generation emission rates for NOX,SOX, time on.The evaluators divided the population of generating
CO2,and mercury using hourly measured emissions data from the units into five groups:those averaging less than 6 hours on,
EPA for the power plants supplying Wisconsin(EPA's`Acid Rain 6-12 hours on,12-24 hours on,24-96 hours on,and more than
Hourly Emissions"data series).The evaluation team aligned its 96 hours on for each time they are dispatched. Marginal emis-
method for estimating emission rates with recommendations of sions in each hour were defined as those produced by the set of
the Greenhouse Gas Protocol initiative(GHG Protocols)developed generating units in the group with the shortest average time on.
by the World Resources Institute(WRI)and the World Business Because the EPA data allow an 8,760-hour accounting of pollutants,
Council for Sustainable Development(WBCSD). Emission factors insofar as energy savings can be assigned to hours of the day and
from reduced use of natural gas at the customer sites were also days of the year,a more accurate emission rate can be estimated
taken from EPA data. by matching the amount of energy savings in a given hour to the
emission rate for that hour. Focus evaluators call this approach
Using the emission rates and evaluation-verified gross electricity time of savings(TOS)emission factors.
savings estimates,the Focus programs together potentially avoided
8.7 million pounds of NOX; 10.7 million pounds of SOX;6.6 billion It should be noted that Wisconsin's power plants are included in
pounds of CO2;and more than 41.5 pounds of mercury from the federal S02 cap-and-trade program(acid rain provisions). In
inception to December 31,2010(see Table 2-23 of the Focus on this cap-and-trade system,S02 emissions may not be considered
Energy Evaluation Annual Report[20101, Revised June 17,2011). reduced or avoided unless EPA lowers the S02 cap.One can say
that the program avoided generation that previously emitted
One implication of adherence to the GHG Protocol is that emission S02, but one cannot claim that future S02 emissions will actually
factor calculations are based on generation data specific to the be reduced due to the effect of the trading program.Starting in
geography of efficiency programs.The relevant set of plants from 2009,the plants were also subject to a cap-and-trade program
which emissions are displaced are those that serve the electric for NOX(the Clean Air Interstate Rule),which has the same effect.
grid in the areas where the efficiency programs are implemented.
A second implication of following the GHG Protocol is that emis- Provided by David Sumi of The Cadmus Group
sion factors are estimated only for plants that are operating on
the margin,that is,the plants most likely to have remained off-
line as a result of a reduction in demand/consumption resulting
from energy efficiency programs.
6-15 www.seeaction.energy.gov December 2012

Chapter 6: Notes
40 U.S. Environmental Protection Agency.(February 2010).Assessing 48 Leakage: In its broadest terms,leakage is the concept that an
the Multiple Benefits of Clean Energy:A Resource for States. activity or outcome expected to occur and remain within a defined
www.epa.gov/statelocalclimate/resources/benefits.html. boundary flows outside the boundary,leading to unintended results.
In energy efficiency programs,an example of leakage is when a
41 Intergovernmental Panel on Climate Change(IPCC).(2007). measure is incented by a program(with the associated costs and
Climate Change 2007:Synthesis Report.www.ipcc.ch/pdf/ assumed savings)but is installed outside of the program's jurisdic-
assessment-report/ar4/syr/ar4_syr.pdf.Contribution of Working tion. In the context of air regulation,such as a cap-and-trade pro-
Groups I,II,and III to the Fourth Assessment Report of the IPCC. gram,an example of leakage is a shift of electricity generation from
Share of different sectors in total anthropogenic GHG emissions in sources subject to the cap-and-trade program to higher-emitting
2004 in terms of CO2-eq. sources not subject to the program.
42"Global Warming Solutions Act:Assembly Bill 32."(2006). 49 Interactive Effects:The influence of one technology's application
pp
California Air Resources Board(CARE).www.arb.ca.gov/cc/ab32/ on the energy required to operate another application.An example
ab32.htm. is the reduced heat in a facility as a result of replacing incandescent
43 U.S. Environmental Protection Agency.(August 2011).Federal lights with CFLs,and the resulting need to increase space heating
Implementation Plans:Interstate Transport of Fine Particulate Matter from another source(usually fossil fuel-fired)or to decrease cooling
and Ozone and Correction of SIP Approvals.www.gao.gov/products/ from another source(usually powered with electricity).
C00127.40 CFR Parts 51,52,72,78,and 97[EPA—HQ—OAR-2009— 50 Acid Rain Program."(2012). U.S. Environmental Protection
0491;FRL-9436-81 RIN 2060—AP50.ACTION:Final rule. Agency.www.epa.gov/airmarkets/progsregs/arp.
44 The Clean Air Act requires state and local air pollution control 51"Clean Air Interstate Rule."(2005). U.S. Environmental Protection
agencies to adopt federally approved control strategies to minimize Agency.www.epa.gov/cair.
air pollution.The resulting body of regulations is known as a State
Implementation Plan(SIP). 52"Regional Greenhouse Gas Initiative."(2012). http://rggi.org/.
45 U.S. Environmental Protection Agency.(2004).Guidance on State 53"California Greenhouse Gas(GHG)Cap-and-Trade Program."
Implementation Plan(SIP)Credits for Emission Reductions from (2012).California Air Resources Board(CARB).www.arb.ca.gov/cc/
Electric-Sector Energy Efficiency and Renewable Energy Measures. capandtrade/capandtrade.htm.
www.epa.gov/ttn/oarpg/t1/memoranda/ereseerem_gd.pdf. For
example,the Metropolitan Washington Council of Governments 54 There are some cap-and-trade program design features that may
developed a regional air quality plan for the eight-hour ozone stan- lead to exceptions to this general rule,including banking and borrow-
dard for the D.C.Region nonattainment area that included an energy ing of allowances,a safety valve(where additional allowances above
efficiency measure.The plan was adopted by Virginia, Maryland, the cap level are made available by the government at a known
and the District of Columbia,and the respective ozone SIPS were price),and establishing the cap at levels above business-as-usual
approved by the U.S. EPA regions in 2007. emissions(so the emission constraint is not binding on emitters).
However,as a first approximation,covered facilities will emit
46 While not directly comparable,another term that is used in the approximately at the cap level.
context of the Clean Air Act is"surplus."The definition of surplus
depends on how the emission reduction will be used,but basically it 55 U.S.Environmental Protection Agency.(September 2005).(Draft).
implies that one cannot double count a program's emission reduc- State Set-Aside Programs for Energy Efficiency and Renewable
tions.As a simple example,if a program's impacts are assumed in the Energy Projects Under the NOx Budget Trading Program:A Review
baseline then one cannot claim credit for the impacts in a SIP. of Programs in Indiana,Maryland,Massachusetts,Missouri, New
Jersey,New York,and Ohio. http://epa.gov/statelocalclimate/
47 World Resources Institute(WRI);World Business Council for documents/pdf/eere_rpt.pdf.
Sustainable Development(WBCSD).(December 2005).GHG Protocol
for Project Accounting(Project Protocol).www.ghgprotocol.org/files/ 56 An example of such a policy is the inclusion of an energy efficiency
ghgp/ghg_project_protocol.pdf.Source of the"assessment bound- resource standard in a State Implementation Plan(SIP).
ary"and"primary/secondary"terminology.
December 2012 www.seeaction.energy.gov 6-16

57"Compilation of Air Pollutant Emission Factors."(2011). U.S.
Environmental Protection Agency.EPA AP-42.www.epa.gov/ttn/
chief/a p42.
58 Intergovernmental Panel on Climate Change(IPCC).(2006).
2006 Guidelines for National Greenhouse Gas Inventories.
www.ipcc-nggip.iges.orjp/public/2006gl/index.html.
59 The exception would be if all of the electricity serving the site of
the energy efficiency activity is generated in a non-grid-connected,
stand-alone generation system.
60 World Resources Institute(WRI);World Business Council for
Sustainable Development(WBCSD).(August 2007).Protocol
Guidelines for Quantifying GHG Reductions from Grid-Connected
Electricity Projects.www.wri.org/publication/guidelines-quantifying-
ghg-red uctions-grid-connected-electricity-projects.
61 The WRI/WBSCD Protocol calls for"conservative"estimates,which
tend to result in underestimating of benefits,versus using a"most-
likely"scenario,which can lead to more accurate estimates.
62 "eGRID."(2012). U.S. Environmental Protection Agency.
www.epa.gov/cleanenergy/energy-resources/egrid/index.html.
eGRID provides a comprehensive source of data on the environ-
mental characteristics of almost all electric power generated in the
United States.
63 The IPM is a multi-regional,dynamic,deterministic linear program-
ming model of the U.S.electric power sector. It provides forecasts
of least-cost capacity expansion,electricity dispatch,and emissions
control strategies while meeting energy demand and environmental,
transmission,dispatch,and reliability constraints.
64'Air Markets Program Data."(2012). U.S. Environmental Protection
Agency. http://ampd.epa.gov/ampd/.See hourly emissions data
from continuous emissions monitoring systems.
65 Mid-Atlantic Regional Air Management Association(MARAMA).
www.marama.org.
66"Promoting Air Quality and Climate Protection with Clean Energy
and Energy Efficiency in the Metropolitan Washington Area."
Metropolitan Washington Council of Governments(MWCOG).
www.mwcog.org/environment/air/EERE/default.asp.
67 California Public Utility Commission(CPUC)selected a team led
by Energy and Environmental Economics, Inc.to model the electric-
ity sector's compliance with AB32,California's Global Warming
Solutions Act.www.ethree.com/public_projects/cpuc2.php
6-17 www.seeaction.energy.gov December 2012

Chapter 7
Impact Evaluation Considerations
Chapter 7 has two major portions.The first several sections cover issues that often arise in impact evaluations:
determining baselines, determining demand savings, calculating persistence of savings, addressing uncertainty of savings
estimates, setting evaluation budgets, and establishing evaluation principles and ethics. Because most of this document
focuses on using evaluation to determine energy (and demand) savings as well as avoided emissions from conventional
efficiency programs,the second portion of this chapter provides brief overviews of the other evaluation objectives:
program feedback, resource planning, and calculating non-energy benefits as well as evaluating some relatively
unique program types.These other program types focus on residential behaviors, education and training, market
transformation, codes and standards, demand response, or greenhouse gas (GHG) mitigation strategies.
7.1 SELECTING BASELINES approaches,the existing conditions are determined generically at
A major impact evaluation decision is how to define the baseline. the time of program design(and as perhaps defined in a technical
Baselines are the conditions,including energy consumption and reference manual [TRM]).
demand,which would have occurred without implementation of the For M&V approaches,the existing condition is typically determined
subject energy efficiency activity.Baseline conditions are sometimes at the time of measure/project installation.The common means for
referred to as business-as-usual conditions and are used to calculate determining such existing conditions baselines,assuming sufficient
project-and program-related savings. Baselines can also include information is available,are through(1)an inventory of pre-retrofit
definitions of non-energy metrics that are being evaluated,such as equipment site-specific characteristics,including nameplate ratings,
air emissions and jobs. and/or(2)an assessment of the existing equipment's or system's
Baselines represent the counterfactual condition that is estimated to energy consumption rates,based on measurements or historical
complete an evaluation.With large-scale consumption data analysis data.Site-specific characteristics include how and when the affected
approaches,the baseline is defined by the characteristics and energy equipment or systems are operated.For example,for an energy
use of the control group(s)used in the analyses. However,for the efficient lighting retrofit,the baseline decisions include the type of
noncontrol group-based impact evaluation approaches,such as lighting equipment that was replaced,the power consumption(watts
deemed savings and measurement and verification(M&V),baseline per fixture)of the replaced equipment,and how many hours the
definitions are determined by the type of project being implemented, lights would have operated.
site-specific issues,and broader,policy-oriented considerations.These
considerations usually result in one of three different types of base DEFINING BASELINE DATA COLLECTION
-
lines:existing conditions,common practice,or codes and standards. REQUIREMENTS
The following is a discussion about selecting appropriate baselines
for the deemed savings and M&V impact evaluation approaches. Assessing baseline and baseline adjustment issues in the
This discussion first defines the three different types of baselines and planning stage is important for determine data collection and
then illustrates how baselines are typically set for common efficiency budgeting requirements.The goal is to avoid reaching the
project classifications. analysis stage of an evaluation and discovering that critical
pieces of baseline information have either not been collected
7.1.1 Existing Conditions Baselines or have been collected with an unreliable level of quality.
As the title implies,existing conditions baselines are what is in place This situation can be guarded against by providing specific
(e.g.,equipment,controls,procedures)at the project site before instructions to program administrators prior to program
the energy efficiency measures are implemented. For example,
for a motor replacement program,it is the energy consumption of implementation. Planning for data collection is necessary to
give administrators notice and justification for collecting data
the motor that was replaced.As another example,for residential they would not ordinarily collect for managing and tracking
behavior-based programs,it is the energy consumption associ-
ated with pre-program behaviors.Typically,for deemed savings program progress.
December 2012 www.seeaction.energy.gov 7-1

7.1.2 Codes and Standards Baseline BASELINES: THE CONSUMER
Energy codes and standards set minimum requirements for energy PERSPECTIVE VERSUS THE PUBLIC
efficient design;as such,they affect energy use and emissions for POLICY
the life of a piece of equipment or building.Codes typically refer to
energy requirements associated with construction of new build- As discussed in this section on baselines,it is typical for
ings and major renovations of existing buildings(e.g., maximum replacement of failed equipment and projects for the baseline
energy use per square feet or minimum insulation requirements). to be defined as common practice or a codes or standards
Standards can refer to buildings or efficiency requirements for requirement.Even projects that are early replacement
specific pieces of equipment such air conditioners and motors(e.g., projects may in subsequent years be evaluated as having
a minimum motor efficiency). a common practice or codes and standards baseline.This
Thus,codes and standards(C&5)baselines are the energy consump- makes sense from a public policy perspective,to not incent
tion associated with buildings or specific pieces of equipment that consumers to buy what they would have normally purchased
meet the legal requirements in place,in the location where a project or what they would be required to purchase,to avoid to over-
is implemented.For example,for a motor replacement program, claiming lifetime savings. However,from a consumer perspec-
the C&S baseline standard might consist of a motor that meets the tive,they are looking at savings from a baseline of what they
minimum requirements of the federal Energy Independence and had before the project was implemented,not what"some
Security Act of 2007(EISA)for new motors.Three nuances associated evaluator says they would have,or should have,done! They
with C&S baselines are as follows: in effect want to see the savings as compared to past energy
• Not all efficiency actions and not all measures are subject to bills,not hypothetical bills.This difference in perspectives
codes or standards;thus,the application of this baseline defini- is important to acknowledge when different baselines are
tion is limited. selected for determining savings from the"policy"perspective
• A variation on a C&S baseline is when a code or standard exists versus the"consumer"perspective,as the savings for each
but is not well complied with;in these cases,one of the other perspective may need to be evaluated differently.
baseline definitions might be more applicable(or a combina-
tion of definitions). In a situation where an applicable C&S exists,the common
• Codes and standards tend to change,and thus C&S baselines practice baseline and the C&S baseline might be the same
need to be regularly updated. value—or not,if the C&S are not complied with or are
7.1.3 Common Practice Baseline too"easy."
Common practice baselines are estimates of what a typical con- • As with C&S,common practices tend to change over time;
sumer would have done at the time of the project implementa- thus,these baselines also need to be regularly updated.
tion.Essentially,what is"commonly done"becomes the basis for 7.1.4 Defining Baselines for Specific Program Types
baseline energy consumption. For example,if the program involves For the purposes of defining baselines,most programs can be
incenting consumers to buy high-efficiency refrigerators that use categorized as follows:
20%less energy than the minimum requirements for ENERGY STAR®
refrigerators,the common practice baseline would be refrigerators • Early replacement or retrofit of functional equipment still
that consumers typically buy.This might be non-ENERGY STAR refrig- within its current useful life
erators,or ENERGY STAR refrigerators,or,on average,something in • Improvement of existing processes(e.g., reducing the energy
between.Common practice is determined by surveys of participants, consumption per-unit of production in a factory)
non-participants,or analysis of market data.The following are three • Replacement of functional equipment that is beyond its rated
nuances associated with common practice baselines: useful life
• If a common practice baseline is selected,the resulting • Unplanned replacement of failed equipment
savings determination probably excludes,on average,savings • New construction and substantial existing building improve-
from free riders. ments(tenant improvements)
• Non-equipment based programs(e.g., behavior-based and
training programs).
7-2 www.seeaction.energy.gov December 2012

TABLE 7.1: Standard Practices for Selection of Baselines for Common Program Categories
PROGRAM •
FOR PURPOSES • •
CONDITIONS STANDARDS
OF
DETERMINATION
Early replacement or retrofit X X X
of functional equipment still Existing conditions baseline C&S baseline for the time Common practice baseline
within its current useful life for the remaining life of period after the remain- for the time period after
Process improvements the replaced equipment or ing life of the replaced the remaining life of the
process equipment equipment
Replacement of functional
equipment beyond its rated X X
useful life
Unplanned replacement for
X X
(of)failed equipment
New construction and
substantial existing building X X
improvements
Non-equipment based X
programs(e.g.,behavior- What people in a control
based and training programs) group would be doing in the
absence of the program
Table 7.1 summarizes standard industry practice for defining base- • The difference between energy use for the old equipment/process
lines for each of these categories of programs. Note that these are replaced(existing conditions baseline)and the new efficiency
not mandates;each jurisdiction and each program should establish measure prior to the time the old equipment/process would
its own baseline scenarios. typically have failed or ceased to be used("initial savings rate").
• The difference between energy use associated with a C&S or
As shown in Table 7.1,the early replacement and process improve- common practice baseline and the new efficiency measure
ment programs present their own challenges in determining the after the time the old equipment or process would have had to
appropriate baselines.This is because the issue of likely remaining have been replaced("long-term savings rate").
life of the replaced equipment(or systems or process)becomes
critically important.In contrast to the other program categories Thus,ideally,early replacement/process improvement programs
listed,early replacement/process improvement programs are geared would be able to take credit for initial savings during the remain-
toward replacing existing(lower-efficiency)equipment with energy ing useful lifetime of the replaced equipment/process.This is the
efficient equipment before the old equipment ceases to function approach that evaluators typically refer to as dynamic baselines or the
or before it would otherwise be replaced.In these early replace- "stair step"approach because of the two levels of savings estimated
ment/process improvement programs,an approach for defining the over the two different periods of time.A key nuance of this dynamic
baseline can actually result in savings being determined based on baseline is that it assumes data on age of the existing equipment can
two different baselines,for two different periods of time:
December 2012 www.seeaction.energy.gov 7-3

be gathered and that the time that a process or piece of equipment reduction is to be reported as part of an evaluation,the term
could be expected to fail and/or simply be ready for replacement must be clearly defined with respect to which time period is
can be reliably estimated.Even so,it is a generally recommended associated with the reduction.
approach for early replacement/process improvement programs. Coincident peak demand savings:the demand reductions that
7.2 DETERMINING DEMAND SAVINGS occur when the servicing utility is at its peak demand from all
(or segments)of its customers.This indicates how much of a
For efficiency programs,determining energy savings is almost utility's peak demand is reduced by the efficiency program.
always a goal of impact evaluations.Energy use and savings are Calculating coincident peak demand requires knowing when
expressed in terms of consumption over a set time period and are the utility has its peak(which is not absolutely known until the
fairly straightforward to define(e.g.,therms of natural gas consumed peak season is over).A term used to describe the relationship
per month or megawatt-hours of electricity consumed over a year). of facility electrical loads to coincident peak demand is diversity
Energy savings results may also be reported by time-of-use period, factor—the ratio of the sum of the demands of an energy
which breaks the year into several periods coinciding with a utility user,or a group of energy users,to their coincident maximum
rate schedule. Examples include peak and off-peak periods of the demand;it is always equal to or greater than 1.0.
summer and winter seasons. DR peak demand savings:the demand reduction associated
In addition,a program's electrical demand savings are also often of
• with a DR program. DR programs reduce a utility customer's
interest and,for some programs,are a primary goal,such as with electricity demand in response to dispatch instructions or price
demand response(DR)programs.Electrical demand savings are signals sent to the program participants—a"call"for reduc-
expressed in terms of kilowatts(kW)or megawatts(MW),which tions.Thus,the DR peak demand savings are determined for
indicate rates of consumption—during a specific period of time.
when there is a"call"for program participants to reduce their
Historically,demand savings(specifically,peak demand savings, energy consumption rate.
rather than simple annual average demand savings)have been Forward capacity market demand savings:(1)the demand
harder to define and determine than energy savings.This is because reduction proposed ("bid")to an electricity system operator to
determining demand savings requires data collecting and analysis meet the level of resource commitments the electricity system
for specific time periods(e.g.,data might be required for summer operator estimates will be needed to meet future peak demand
weekdays between noon and 6 p.m.)compared with just need- on the system,and(2)the actual demand reduction that occurs
ing aggregated monthly energy data.However,with technology once a commitment is made.68(See the sidebar on page 70,
advances lowering the cost of meters,the spread of"smart meters," the Independent System Operator of New England [ISO-NE]
sophisticated wireless sensors,and related software,it is becom- program.)
ing easier to cost-effectively collect the data needed to calculate
demand savings. The calculation for demand savings isstraightforward—whether for
a year,a day,or a specific 15-minute period of time:
The first step in determining demand savings is defining the specific
metric of interest(i.e.,how demand savings are defined).The follow—
ing are common demand savings definitions that indicate the specific Each of the impact evaluation approaches described in Chapter 4,to
time period of interest for determining demand savings. varying degrees of accuracy and with varying degrees of effort,can
• Annual average demand savings:total annual energy savings be used to determine demand savings using the above equation.The
divided by the hours in the year(8,760). In the Northwest "trick,"as mentioned above,is to collect the energy savings data for
United States,this is termed average MW,or MWa.Similarly, the intervals of interest(i.e.,the time period in the above equation).
average monthly demand savings or average daily demand If annual average demand savings is the only metric of interest,
savings may be determined. then only annual energy savings data are necessary. However,if
• Peak demand savings:several definitions are used;all involve peak demand reduction,coincident demand reduction,or demand
determining the maximum amount of demand reduction response peak demand reduction values are desired,then hourly
during a "peak" period of time,whether that is annual, or 15-minute(a typical period of demand recording)energy sav-
seasonal,or a specific period such as during summer weekday ings data,or estimates,are required for at least those specific time
afternoons or winter peak billing period hours. If peak demand periods of interest.
7-4 www.seeaction.energy.gov December 2012

ISO-NE M&V MANUAL FOR WHOLESALE FORWARD CAPACITY
In 2007,the Independent System Operator of New England Baseline conditions.The manual specifies baseline condi-
(ISO-NE)developed an M&V manual that describes the mini- tion requirements for failed equipment(codes/standards
mum requirements the sponsor of a demand resource project or standard practice,whichever is more stringent),early
must satisfy to qualify as a capacity resource in New England's retirement(codes/standards or measured baseline),and new
wholesale electricity forward capacity market(FCM). Demand construction(codes/standards or standard practice).Where
resources eligible to participate in FCM include demand response, standard practice is used,baseline conditions must be docu-
emergency generation,distributed generation,load management, mented and meet the confidence and precision requirements.
and energy efficiency.They are eligible to receive a capacity For distributed generation and emergency generation,the
payment($/kW per month)based on the measured and verified baseline is zero.The baseline for real-time demand response is
electrical reductions during ISO-specified performance hours. calculated using a modified rolling average of the host facility
The manual was developed with input from key stakeholders in load on non-event weekdays during the same hours as the
the region,including members of the New England Power Pool, called event.
ISO-NE,the New England state regulatory staff,electric utility Measurement equipment specifications.The project sponsor
program administrators,Northeast Energy Efficiency Partnerships, must describe measurement,monitoring,and data recording
and energy service,consulting,and technology providers.The device type that will be used(and how it will be installed)for
manual specifies the minimum requirements a project sponsor's each parameter and variable.Any measurement or monitoring
M&V plan must address,including the following: equipment that directly measures electrical demand(or proxy
• M&V methods.The sponsor must choose from options variables such as voltage,current,temperature,flow rates,
largely based on the IPMVP options A through D(or and operating hours)must be a true root-mean square(RMS)
equivalent). It should be noted that ISO-NE deviates from measurement device with an accuracy of at least±2%.
IPMVP guidance in particular for Option A,in an attempt to Monitoring parameters and variables.The project sponsor
incorporate use of deemed savings values as an acceptable must describe variables that will be measured,monitored,
approach for efficiency programs,given that the M&V counted,recorded,collected,and maintained,and meet
manual does not explicitly allow for use of deemed savings minimum requirements for data to be collected by end-use
as a"methodology,"as provided in this guidance document). and monitoring frequency.
The ISO-NE manual also allows for other M&V techniques to
be used in combination with one or more of these,including The PJM Interconnection subsequently developed and adopted
engineering estimates supplemented with data collected on an energy efficiency M&V manual in 2009,building largely on the
the equipment affected by the measures,and/or verifiable ISO-NE M&V Manual.These two documents have largely served
measure hourly load shapes(which must be based on actual to inform the development of wholesale M&V standards and retail
metering data,load research,or simulation modeling).All M&V model business practices for energy efficiency by the North
demand resources,including distributed generation and American Energy Standards Board(NAESB).
emergency generation,must be metered at the generator.
Provided by Julie E.Michals,Director,Regional EM&V Forum at Northeast Energy
• Confidence and precision.The project sponsor must Efficiency Partnerships,Inc.For more information,see the ISO-NE and PJM EE M&V
describe a method for controlling bias(e.g.,calibration of manuals,respectively,at www.iso-ne.com/rules proceds/isone mnls/index.html
measurement tools,measurement error,engineering model) and www.pjm.com/committees-and-groups/closed-groupsleetf..ospx.
and achieving a precision of±10%,with an 80%confidence
level(two-tailed test)around the total demand reduction
value.This requirement also applies to precision level for
statistical sampling.
December 2012 www.seeaction.energy.gov 7-5

Ideally,evaluation results would indicate 8,760 hours(or about rates,such as those for a lighting controls/daylighting program or
35,000 15-minute periods)per year of energy savings data that could a program that involves installation of variable-speed motor drives
be easily translated into demand savings. However,in practice,both on building ventilation fans.In the latter case,there might be zero
primary and secondary methods for determining demand savings (or even negative)savings when the fan is at full speed(which often
are used.Primary methods involve the actual collecting of hourly, occurs during peak period hours).
15-minute,or even"continuous"demand data during the periods of
interest—for example,during the peak hours of the summer months 7.3 PERSISTENCE OF SAVINGS
(peak season)of each year. One important evaluation issue is how long energy savings are
expected to last(persist)once an energy efficiency activity has taken
Sources of hourly or 15-minute data include facility interval-metered place.While energy and demand savings are often reported with
data,time-of-use consumption billing data,monthly billing demand respect to just first-year savings,the longer a measure,project,or
data,and field-measured data.When interval or time-of-use program provides savings,the more valuable it is. In addition,a proj-
consumption data are available,they can be used in regression ect with a 5-year payback will not be of much value if the efficiency
analyses to account for the effects of weather,day type,occupancy, measures only last three years.Thus,estimates of total or lifetime
and other pertinent change variables to determine the demand savings are important for determining the cost effectiveness of an
savings caused by the subject program.Of course,hourly demand efficiency action,as well as their value as a long-term resource.
data can require hourly independent variable data(e.g.,weather)for
proper regression analysis. 7.3.1 Definitions
Secondary methods apply load shapes to collected energy A persistence study measures changes in program impacts over time.
consumption data that are only available as averaged or total values Related to these studies are measure-retention studies that assess
on a daily,monthly,or annual basis. Load shapes indicate energy (1)the length of time the measure(s)installed during the program
consumption per hour.These load shapes,for whole facilities or by
year is maintained in operating condition,and(2)the extent to
end use(e.g.,lighting,cooling,or heating),may be available from
which there has been a significant reduction in the effectiveness of
studies of related programs in similar markets.One source for the the measure(s).Two related terms and their definitions are helpful
data is the energy savings load shapes,by measure,included in the
to know:
California Database for Energy Efficiency Resources(DEER).69 Measure persistence:the duration of an energy consum-
ing measure,taking into account business turnover,early
For example, load shapes might exist for typical weekdays and retirement of installed equipment,technical degradation
weekend days in the form of hourly percent of monthly energy use factors(s),and other reasons measures might be removed or
for an office building. In this example,if the lighting system retrofit in discontinued.
a building saved 3,000 kWh in a month,and the load shapes indicate Effective useful life(EUL):an estimate of the duration of
that the lighting system energy use in a building during the hour of savings from a measure. It is estimated through various means,
interest(say 4 p.m.-5 p.m.on weekdays)is 1%of the monthly total, including the median number of years that the efficiency
then it is assumed that 1%of the energy use is saved during that measures installed under a program are still in place and
peak hour-30 kWh,or a peak demand reduction of 30 kW.Another operable. It is also sometimes defined as the date at which 50%
example of secondary data is if peak-to-average demand ratios for of installed units are still in place and operational. EULs are
lighting systems in the building types of interest are available.So,as also defined as"measure life,"which is essentially the length of
another example,for this building that saved 3,000 kWh in a month, time that a measure is expected to be functional.
or an average of about 4.2 kWh per hour,if the lighting peak-to-
average demand ratio is 5:1,then the estimated peak demand Paraphrasing from a recent conference paper,70 more than 100
savings is 21 kW. energy efficiency persistence-type studies have been conducted,
examining in-situ median lifetimes for residential and non residential
These load shape approaches can be relatively effective for measures measures.A review of results from measure-based EUL studies
with constant performance,such as a lighting retrofit that reduces around North America showed that measure lifetimes for the type
energy consumption as a percentage of baseline energy use(i.e., of measures normally offered in prescriptive programs(programs
change-out of 200 W fixtures for 100 W fixtures). However,these with defined measures and incentives for implementing the mea-
approaches do not work as well with projects with variable savings sures)and installed in typical end uses are fairly consistent for many
7-6 www.seeaction.energy.gov December 2012

measure-based programs in commercial,residential,and industrial the pre-project level of service is very low,for example,in low-
sectors.71 A key factor affecting the quantity of savings being deliv- income housing where prior to the efficiency actions,heating and
ered from program-related installations of equipment is whether the lighting systems were barely functional,but the new systems allowed
measures perform at the new efficiencies consistently over time,or the occupants to adequately light their homes and heat them to
whether their efficiency performance degrades over time. Decays in comfortable levels.
net technical performance can be an important issue,particularly for
measures for which savings accrue over long periods of time. The issue for impact evaluation is whether rebound is explicitly or
implicitly included in the savings determination.An example of an
As discussed throughout this guide,energy savings achieved over explicit consideration is the use of a deemed rebound effect factor,
time are a"difference"from a baseline performance.Thus,persistence a form of a non technical degradation factor.As with all deemed
looks at degradation patterns that would be realized in standard factors,it should be specific to the applications associated with the
efficiency equipment,or typical(non-efficiency)consumer behaviors, subject program and based on actual historical data.Another explicit
and compares them with the degradation patterns of the program's approach would be a long-term study of rebound in the participants
efficient equipment or behaviors.Therefore,savings are the differ- and a control group of non-participants.Current standard energy
ence over time between the energy use of the efficient equipment or efficiency evaluation practices do not use either of these approaches
behavior and the standard equipment or behavior it replaced—with or any other explicit approaches for assessing rebound.
consideration of both baseline and project equipment/behavior
degradation in performance(which may be the same). What is used commonly,though,is an implicit approach,and
although it is not often acknowledged as including rebound effects,
7.3.2 Rebound it does.The implicit approach is associated with a savings determina-
Although it is not usually included in persistence determinations, tion method that includes savings determination based on both
the rebound effect(also called take back,snap back,backfire,and the project period(after the efficiency project is implemented)and
a few other terms)resurfaces as an issue periodically,as does its pre-project service levels.This is true with the large-scale consump-
potentially negative impact on energy savings proponents.A form of tion data analysis approaches discussed in Chapter 4.If the program
this concept was first put forth by British economist William Jevons participants increase their service level and energy use compared
in 1865.An interpretation of the"Jevons paradox"is that increases with a control group of non-participants,the calculations will capture
in efficiency of coal processes would cause coal consumption to this in the savings determination.
increase,to a level that would exceed previous consumption levels.
Jevons paradox was based on the industrial sector,where coal costs There is no"correct"answer to whether or not rebound should be
where a high percentage of operation,and he did not consider the evaluated,particularly given that the magnitude of the rebound
counterfactual,or what would happen if the efficiency of processes effect is not known;however,the evaluation planning efforts and the
did not increase.In any case,Jevons,fundamental argument has reported results should indicate whether rebound is addressed or not.
been repackaged and repurposed,correctly or incorrectly,to cover
micro-level and macro-level impacts;it has been extended to 7.3.3 Determining Persistence
questions about energy efficiency(and carbon emissions reduction) Persistence studies can be expensive,and are thus not often
efforts in general,and been given the general term"rebound." In undertaken for individual programs. Past experience indicates that
this guide,the rebound effect is defined as a change in energy-using long periods of time are needed for persistence studies,and there
behavior that yields an increased level of service accompanied by are significant challenges to conducting these studies,such as long
an increase in energy use that occurs as a result of taking an energy lifetimes of measures(making it impractical to wait for measure
efficiency action. failures or consistent patterns of degradation),incomplete data sets,
high cost of data collection,and of course,the need for trained staff.
Few,if any,in the efficiency industry argue that the rebound effect
never occurs,although few rigorous studies of rebound have been The most accurate manner to report savings persistence is probably
completed.72 Clearly,in some situations,an energy efficiency action as an annual percentage(e.g., 100%in year 1,98%in year 2,95%
results in an increased level of service and an associated increase in in year 3,and so on).However,the most common way that persistence
absolute energy use greater than would be experienced if the level is indicated is via a value for EUL(e.g.,20 years for an energy-efficient
of service did not increase.This is most probable in situations where refrigerator).
December 2012 www.seeaction.energy.gov 7-7

The basic approaches for assessing persistence are as follows: the difference between(1)actual energy consumption and(2)what
• Use of historical and documented persistence data,such energy consumption would have been if the efficiency measures had
as manufacturer's studies or studies done by industry not been implemented is an estimate of energy(and demand)sav-
organizations such as the American Society of Heating, ings.While program evaluations seek to reliably determine energy
Refrigerating,and Air-Conditioning Engineers(ASHRAE) and demand savings with reasonable accuracy,the value of the
• Laboratory and field testing of the performance of energy- estimates as a basis for decision making can be called into question
efficient and baseline equipment if the sources and level of uncertainty of reported savings estimates
are not understood and described.While additional investment in
• Field inspections,over multiple years,of efficiency activities the estimation process can reduce uncertainty,trade-offs between
that constitute the subject programs evaluation costs and reductions in uncertainty are inevitably
• Non-site methods such as telephone surveys and interviews, required;thus,improved accuracy(and associated EM&V costs)
analysis of consumption data,or use of other data(e.g.,data should be justified by the value of the improved information.
from a facility's energy management system)
Uncertainty is a measure of the"goodness"of an estimate.Without
The 2006 California Evaluation Protocols contain a complete section some measurement of uncertainty,it is impossible to judge an
on persistence analyses and can be used to learn more about this estimate's value as a basis for decision making. Uncertainty refers to
subject.73 the amount or range of doubt surrounding a measured or calculated
value.Any report of gross or net program savings,for instance,has a
There are two specific nuances of persistence analyses: halo of uncertainty surrounding the estimated values relative to the
• Savings can live as long as the behavior that enables the true values(which are not known).As defined this way,uncertainty
efficiency is continued.This can be well past the life of the first is an overall indicator of how well a calculated or measured value
purchase of that type of technology,or the opposite—well represents a true value.
less than the technical life of a measure,if it is not being used.
Thus,it is advisable to evaluate persistence of measures associ- Uncertainty of savings-level estimates is the result of two types
ated behaviors(e.g.,continuing to use a new control system of errors:
or always buying the efficient product when replacement is • Systematic errors:errors that are subject to decisions and
required or responding to information programs)rather than procedures developed by the evaluator and are not subject to
just the effective useful life of a technology. chance,also called bias.
• As discussed in Section 7.1 on baselines,one baseline option is • Random errors:errors occurring by"chance."One important
to use dynamic baselines for early replacement programs.The type of random error is the changes in energy use that can be
classic example is a motor with five years of remaining life being due to unobserved influences(i.e.,unobservable independent
replaced with a new efficient motor.For the first five years of the variables such as different preferences among people within
new motor's effective useful life,the baseline is the motor that a population).Additionally,whenever a sample is selected to
was replaced.However,for the remaining years of the EUL,the represent the population—whether the sample is of appliances,
baseline would be a common practice or code/standard compli- meters,accounts,individuals,households,premises,or organiza-
ant motor,as of year five—when it is assumed that the original lions—there will be some amount of random sampling error.
motor would have been replaced.Some may ask whether this Any selected sample is only one of a large number of possible
correction in year five is a persistence correction or a baseline samples of the same size and design that could have been
correction.Whichever way this may be classified,baseline or selected from that population.For each sample,values calculated
persistence,it is important to transparently report measure will differ from the other potential samples simply because of the
savings over the lifetime of the motor(and program)with an element of chance in choosing a particular sample.This variability
indication that a dynamic baseline was used if such is the case. due to chance factors(the"luck of the draw")is termed random
7.4 CONTROLLING UNCERTAINTY74 sampling error.Random error can be reduced by increasing the
sample size.
As discussed in Chapter 3,a significant challenge in evaluating energy
efficiency programs is the impossibility of direct measurement of Because uncertainty arises from many different sources,it is usually
the primary metric—energy(and demand)savings.Consequently, difficult to identify and quantify the effect of all potential sources.
The distinction,described above,between systematic and random
7-8 www.seeaction.energy.gov December 2012

sources of error is important because different procedures are program,are different than the rest of the population.As such,
required to identify and mitigate each.Evaluation reports often it may be very difficult to find an appropriate control group to
identify only uncertainty arising from random error,because this use;thus,evaluation findings may potentially produce biased
source of error is usually the easiest to quantify using confidence results.Other forms of self-selection bias include those who
intervals and statistical significance tests that are available to provide respond to surveys versus those who do not.
quantitative estimates of uncertainty caused by random error(see 4. Analysis(modeling).Estimates are often created through
sidebar on statistical terms). statistical models.Some are fairly simple and straightforward
On the other hand,uncertainty attributable to systematic errors does (e.g.,estimating the mean)while others are fairly complicated
not have a single comparable,quantitative measure.Rather,sources (e.g.,estimating response to temperature through regression
of systematic error/bias are specific to individual studies,depending
models).Regardless,errors may occur due to using the
on equipment used,research staff,or research and data collec- wrong model,creating the wrong counterfactual,assuming
tion procedures employed.To assess uncertainty from systematic inappropriate functional forms,including irrelevant information,
sources/bias,it is necessary to address the rigor of data collection,
or excluding relevant information.For example,in modeling
measurements,and analysis.Thus,in summary,uncertainty is energy use of air conditioners,the evaluator may only use
typically calculated and reported through the objective analysis of cooling degree days as the independent variable if home type
random errors and the subjective analysis of systematic errors. or square footage is not available.Thus,the statistical model
will attribute all the observed differences in energy use to
7.4.1 Systematic Errors/Bias temperature,while clearly a portion of the use is attributable to
Systematic errors are problematic because they cause a savings the home size.This model will introduce systematic error.
estimate to be biased. Bias is simply the extent to which a measure-
ment or analytic method systematically underestimates or overesti- For the measurement-related sources of bias,the best solutions
mates a value.Systematic errors potentially occur from the way data are sound engineering and analysis practices such as calibration of
are measured and/or collected,and/or the way in which analysis meters,use of experienced and trained personnel;use of rigorous
(and/or modeling)is conducted.Some specific examples of sources data analysis,and development and rigorous application of quality
of bias are below. assurance and control procedures.With respect to bias associated
1. Measurements.At times,equipment used to measure with data collection and analysis,implementing the proper random
consumption may not be completely accurate. Human errors selection approaches is very important for avoiding bias and mitigat-
(e.g.,errors in recording data) may also cause this type of error. ing systematic errors in a savings estimate.
Measurement error is reduced by investing in more accurate 7.4.2 Random Errors
measurement technology and training and by more accurately There are many possible reasons why energy use might change
recording and checking data. In terms of specific instruments, for any program participant's facility in any particular month
the potential magnitude of this type of error is provided by (e.g.,a household may take a vacation,a commercial office may
manufacturer's specifications. In most applications,this error add employees,a factory might have higher or lower production or
source is ignored, particularly when data sources are utility- change product mix,a school might be in a location with an unusually
grade electricity or natural gas meters. However,other types of cold month).An evaluation of a program's impact tries to tease
measurements,such as flow rates in water or air distribution out only the changes in energy use that are due to the program,as
systems,can have significant errors. opposed to the other factors.While some of the other factors may
2. Data collection.If some parts of a population are not included be easily observable(e.g.,occupancy or weather),many factors(e.g.,
in the sample,non-coverage errors result.This may cause a vacations)are not.
problem because the value calculated from the sample will not
accurately represent the entire population of interest. Therefore,when change in energy use is estimated,there is some
3. Self selection.Other sources of bias include using a control chance(hopefully,a very high chance)that the actual program impact
group that is not comparable with the treatment group of is being measured. However,there is also some chance that other,
participants. For example,in programs where participation is unobserved factors happened to cause the change in energy use.
voluntary,a potential source of bias is what is referred to as There is therefore some chance that the evaluation erroneously
"self-selection." Participants,by simply choosing to be in the estimates that the program caused the change in energy use when in
fact it was caused by other factors(or combinations of the program
December 2012 www.seeaction.energy.gov 7-9

and other factors).The same is the case when indicators other than convenient shorthand for expressing the interval believed to contain
energy use are being determined as a precursor to determining the actual value.The confidence level is the probability that the inter-
energy savings(e.g.,changes in operating hours of lighting systems val actually contains the target quantity.For example,if the savings
that have had occupancy sensors installed). estimate is 530 kWh,and the relative precision level is 10%,then the
confidence interval is 530±53 kWh).75
One specific source of random errors that is often a concern is
that evaluators typically do not have access to an entire popula- Put another way,if a project with a 10%(of baseline energy use)
tion of interest,either because the population is too large or the savings estimate with a 20%precision at the 90%confidence level
measurement process is too expensive or time-consuming to allow means that the true program energy savings is between 8%and 12%
more than a small segment of the population to be observed.As a of baseline energy use,with 90%probability.It is very important to
result,they must base their decisions about a population on popula- note that this indication of savings certainty is based on analysis of
tion sample data,resulting in sampling errors(one of the major random error and does not address systematic errors.
sources of random errors). Note that even if the entire population of
interest(e.g.,every household in a program)is observed,there can One way to interpret confidence intervals and precision is as a
also be random error due to unobserved variables. measure of risk. For example,with an estimate of 10%savings,
there is some risk that the true savings are more or less than 10%;
The following are examples of energy efficiency program impact indications of confidence and precision can help quantify that
evaluation samples: risk. In general,high levels of confidence that the estimated value
• Residential efficiency retrofit program.A sample of homes is falls within the predicted interval can be achieved with wider(less
selected for analysis(versus all of the homes that were retrofit- precise)intervals,while narrower(more precise)intervals permit
ted).The sample may be organized into homes with similar less confidence. In other words,when all else is held constant,there
physical characteristics,similar occupants,similar vintages,or is a trade-off between precision and confidence.76 As a result,any
other similarities. statement of precision without a corresponding confidence level is
• Commercial building lighting retrofit program.A sample of the incomplete.
"spaces"(e.g.,offices,hallways,common areas)is selected for For example,suppose the average savings among participants is
determining average operating hours of the lights in each space. estimated to be 1,000 kWh per year,and the analyst determines this
• Industrial motors retrofit program.A sample of motors that estimate to have 16%relative precision at the 90%confidence level.
were installed is selected for metering of power draw during a The same data set and the same formulas may be used to estimate
range of operating conditions and time periods. 10%relative precision at the 70%confidence level.Without report-
• New construction building incentive program.All of the build- ing the confidence level,the second uncertainty expression seems to
ings in a program are selected for analysis,but only within a have less uncertainty when,in reality,they are identical.In reporting
certain time period(e.g.,one month per year). estimates from a sample,it is essential to provide both the precision
• Net savings analysis of participants in an efficiency program. and its corresponding confidence level(typically 80%to 90%for
A sample of participants and a sample of non-participants are energy efficiency evaluations).
selected for interviews. Before concluding this section on controlling uncertainty,two other
Random error as a whole can be estimated by using the laws of terms are introduced:internal validity and external validity.The
probability.In other words,the potential magnitude of the random above discussion is addressing internal validity—that is,an evalu-
error for any value calculated from a sample can(usually)be estimated. ation is internally valid if the observed results are known to have
Sample size can be a particularly important aspect of an evaluation been caused by the program as opposed to other factors that may
design,and decisions about the sample size are one of the key influ- have influenced the outcome(i.e.,the estimate is unbiased and
ences on the random error and the overall uncertainty of the evalua- precise).On the other hand,an evaluation is externally valid if the
tion.A larger sample size will increase the statistical significance of the observed outcomes can be generalized and applied to the population
estimate,which will reduce the chance of the random error. from which the sample was selected,as well as new populations,
circumstances,and future years.The external validity of an evalua-
The common factors for reporting random error-associated tion is an important consideration if the evaluation results are to be
uncertainty are confidence and precision.Precision provides applied to other programs(without their own potential studies or
7-10 www.seeaction.energy.gov December 2012

evaluations)operating in different years,locations,and/or popula- evaluation requires consideration of all of the aspects of the evalua-
tions;for example,external validity answers the question of whether tion process,particularly consideration of 13 of the 14 issues raised
the results of a 2010 Oregon CFL evaluation can be applied to a 2012 in Chapter 8 on planning(the last issue is evaluation scale and budget
version of the Oregon CFL program. itself).This section,however,discusses budgeting in the context
of managing the risks of savings uncertainty.It also provides some
In conclusion,evaluation of savings uncertainty is an ongoing process information on the budgets that different states have assigned to
that can consume time and resources.It also requires the services evaluation activities,as an indication of the range of typical budgets.
of evaluation contractors who are familiar with data collection and
analysis techniques.And,of course,reducing errors usually increases 7.5.1 Using Impact Evaluations to Manage Risk
evaluation cost.Thus,the need for reduced uncertainty should be The evaluation process must find the right balance between interest
justified by the value of the improved information.That is,is the for the most accurate estimates possible with the costs for obtaining
value worth the extra cost,and are the evaluation activities them- such accurate estimates.This leads to a basic impact evaluation
selves cost effective? question:"How good is good enough?"Asking"How good is good
enough?"is a short version of asking"How certain does one have
7.5 EVALUATION BUDGETS: BALANCING to be of the energy savings estimate that results from evaluation
THE VALUE OF EVALUATION RESULTS AND activities,and is that level of certainty properly balanced against the
UNCERTAINTY amount of effort(e.g.,resources,time,money)it takes to obtain
This section provides input on establishing a budget for an impact that level of certainty?"The implication is that energy efficiency
evaluation.Establishing a budget(i.e.,setting the funding level)for an investments should be cost effective and evaluation investments
STATISTICAL TERMS
Census(total enumeration)consists of collecting data from each that is the case.Alternatively,one may be 99%confident that the
and every unit in the population(e.g.,metering every replaced light true average savings are within±50%of the estimate of 1,000 kWh.
bulb).Sampling only chooses a small part of the units from
the population for data collection and analysis. If the estimated outcomes are large relative to the variation,they
tend to be statistically significant.On the other hand,if the amount
For any value calculated from a sample,a set of descriptive of variability is large relative to the estimated outcome,one is
statistics(such as the mean,standard deviation,standard error,and unable to discern if observed values are real or simply random.In
a confidence interval)can be calculated.Standard deviation is a other words,when variability is large,it may lead to precision levels
measure of variability showing the extent of dispersion around the that are too large(e.g.,more than±100%)for observed estimates
mean.In normally distributed data,about 68%of observations are (e.g.,estimated savings)to be meaningful.In an extreme example,
within one standard deviation of the mean.Based on the amount if the observed average is 1,000 kWh and the associated precision
of variability and standard deviation,a confidence interval can is±150%,true average savings are somewhere between a negative
be calculated. 500 kWh(which means the measure actually caused consumption
to increase)and 1,500 kWh.
To communicate evaluation results credibly,outcomes need to be
expressed with their associated variability.Confidence refers to To formalize these relationships,evaluators use a test called the
the probability that the estimated outcome will fall within some t statistic.The t statistic is a measure of how reliable a statistical
level of precision.Statement of precision without a statement of estimate is.A t test produces a critical value;for example,at a
confidence proves misleading,as evaluation may yield extremely 90%level of confidence,the critical value of t is 1.645.When
high precision with low confidence or vice versa.For example, the parameter estimate,such as the mean kWh savings,is small
after metering a sample of affected equipment,one may estimate relative to its associated variability,the t statistic value is low.When
average savings as 1,000 kWh.This is an estimate of the true aver- using a 90%confidence level,if the t statistic is less than 1.645,the
age savings.Further,one may able to state that the true average is evaluator concludes the estimated value(e.g.,mean kWh savings)
within±1%of the estimate(precision),but only be 30%confident is not reliable:it could be negative,positive,or zero.
December 2012 www.seeaction.energy.gov 7-11

should consider risk management principles,and thus balance 7.5.2 Budget-Setting Considerations
the costs and value of information derived from evaluation(i.e., With respect to setting an evaluation budget,the primary challenges
evaluation should also be cost effective).Impact evaluation is about are typically balancing(1)the cost,time,and effort to plan and
managing risk. complete the evaluation(s);(2)the uncertainty of various impact
evaluation approaches;and(3)the value of the information gener-
Conceptual approaches that draw upon risk management techniques ated by the efforts.Conceptually,this is shown in Figure 7.1,where
provide a useful structure for addressing evaluation issues identified the goal is to find the balance point between increasing incremental
in this guide.Unfortunately for energy efficiency,risk management investments in evaluation(costs)and decreasing incremental value
is hampered by the large number ofdifficult-to quantify aspects of in the evaluation information.
efficiency and evaluation,although the tools for addressing these
difficulties are improving.Energy supply-side resources have uncer- Most of the value of impact evaluation information is tied to the value
tainty and risks as well(e.g.,uncertainties associated with future fuel of energy savings and overall program integrity.In general,low-risk
costs). However,perhaps the single most identifiable risk of efficiency projects require less evaluation confidence and precision;high-risk
is the inability to directly measure savings,which creates uncertainty. projects require more confidence and precision.The acceptable level
of uncertainty is often a subjective judgment based on the value of the
Tolerance for uncertainty is driven by how much risk is associated energy and demand savings,the risk to the program associated with
with getting the wrong answer.For example,with energy efficiency, over or underestimated savings,and a balance between encouraging
the risks include crediting too much or too little savings to the efficiency actions and high levels of certainty.
actions that have been taken to comply with an energy efficiency
program objective or mandate,such as an energy efficiency resource Thus,an important aspect of evaluation planning is deciding what
standard.This can lead to expending too many resources on ineffec- level of risk is acceptable and determining the requirements for
tive actions(or the opposite),or simply not obtaining the desired accuracy and a corresponding budget.How much risk is acceptable is
outcome(i.e.,less energy consumption). However,there is another usually related to many factors,which raise the following questions:
counterbalancing risk if policymakers minimize energy efficiency as • How large is the program in terms of budget and savings goals?
an eligible resource because of this measurement risk.This could Larger programs tend to have larger evaluations but smaller
take the form of spending too much on EM&V beyond the need for evaluation costs as a percentage.
lowering uncertainty and, more important,trying to eliminate this What is the level of uncertainty associated with the expected
measurement risk by excluding viable(or promising but difficult savings of a program,and what is the risk that the program
to"prove")efficiency resources that are replaced with supply-side poses in the context of achieving(or not)portfolio-savings
resources that have different,and perhaps greater,risks associated goals( g•,oa e. are the subject program's projected savings 10/° or
90
with their performance and/or lifecycle costs. °%of the portfolio savings goal)?
An important aspect of risk management is always the relative risk • Does the savings determination need to indicate how much
(the`As compared to what?"question),which for efficiency includes energy was saved,or just that the savings were above a certain
comparing the measurement risk with the risks associated with level?The latter is usually easier to determine than the former.
supply-side resources,such as future estimates of fuel costs.Thus, • Is it a new program with uncertain savings or an established pro-
another risk to be managed with efficiency evaluation is to avoid gram with well-understood savings?Established programs with
requirements that over-specify certainty and/or the use of overly a history of well-documented savings may not require the same
conservative versus most-likely assumptions,which can result in level of evaluation that a new program with no history requires.
unnecessarily excluding or limiting energy efficiency as an energy Related to this consideration is how much confidence exists in
resource strategy. pre-program,projected savings estimates. If a fair amount of
Perhaps these two quotes attributed to Albert Einstein best capture
effort has gone into feasibility studies and perhaps pre-testing
the essence of conducting evaluations: (e.g.,pilots),then less of an evaluation effort may be required.
• "Everything should be as simple as it is,but not simpler." Is it adequate to simply verify that the individual projects in
a program were installed(and perhaps operating correctly)?
• "Everything that can be counted does not necessarily count; Or,on the other end of the cost spectrum,are rigorous field
everything that counts cannot necessarily be counted." inspections,data collection,and analyses on all,or a large
sample of, projects in a program required?
7-12 www.seeaction.energy.gov December 2012

FIGURE 7.1: Incremental value of information versus incremental cost of evaluation
I
Value of Improved Certainty Value of Improved Certainty
0 > Cost of Improved EM&V < Cost of Improved EM&V
Q
O
LL
Z
LL
O
W
>J Balance Point of
I Diminishing Returns
J
H
Z
W
Z
INCREMENTAL COST
• Is the program likely to be expanded or contracted?A program savings needs to be.As discussed earlier in this chapter,there
that may be expanded(i.e., increased in goals and budget) are different ways to calculate and report demand savings,with
probably deserves more analyses to confirm whether it should very different levels of effort required.
be expanded than one that is likely to receive no additional What are the reporting requirements and who must review
funding or n be cancelled. (and approve)evaluation results?While all evaluations should
• How long has it been since the last evaluation of the subject have well-documented results,the frequency that savings need
program,and has the program changed in the interim? to be reported,and to what audience(e.g.,a regulatory body)
• Do savings need to be attributed to specific projects within a can influence the scale of the effort and budget.
program(e.g.,a custom retrofit program where each project is Are avoided emissions also to be determined,and will the
unique)?If savings values for each project are desired,then a avoided emissions benefits be used in a regulatory program?
census evaluation is probably required.This is more costly than As discussed in Chapter 6,emissions can be calculated simply
evaluating a sample of projects. or with significant effort and accuracy. If avoided emissions
• How long,in months or years,does the evaluation need to be values will be used in a regulated program,the analyses may be
conducted?Longer evaluation cycles require more funding.On subject to specific requirements and third-party verification.
the other hand,evaluations that have started late,and might • Are other non-energy benefits to be evaluated and quantified?
be in a start-up mode,will require smaller budgets(and prob- If this is more than an anecdotal exercise,then additional
ably deliver less detailed results). resources will be required.
• What is the time interval for reporting savings?For example, • Are the savings reported to be used as"information only"or is
reporting annual or monthly savings estimates is usually much there a regulatory or statutory requirement associated with the
simpler than reporting hourly savings.This is particularly use of the impact evaluation results?For example:
important when deciding how accurate an estimate of demand
December 2012 www.seeaction.energy.gov 7-13

— Is there a savings goal,perhaps associated with an en- report,shows total EM&V spending for program administrators who
ergy efficiency resource standard,for which the impact responded to their survey in the United States and Canada.Looked
evaluation will determine compliance? at in total,for natural gas and electricity programs,the average
— Are there cost-recovery and/or lost revenue recovery percentage of program budget spent on EM&V in 2011 was about
financial impacts for utilities to be decided upon by a regu- 3.6%.This is consistent with the range of EM&V budgets for both
latory body using the impact evaluation results? program administrator efforts and independent third-party evaluator
— Does the program administrator have a performance-based efforts that are cited in Tables 7.3 and 7.4.
incentive,the monetary value of which will be determined Table 7.3,from a 2010 survey report of evaluation practices and
on the basis of the impact evaluation reported metrics? issues in the United States,shows total EM&V 2008 and 2009 bud-
Budgets may also be influenced(increased)to accommodate gets for a sample of states.Table 7.4,from the same survey report,
follow-up studies aimed at assessing and reducing measurement
indicates additional information on EM&V budgets allocated among
error,or to pay for additional short-term metering,training of staff,
three main types of EM&V activities(process,impact evaluations,
or testing of questionnaires and recording forms to reduce data and market research).
collection errors.The determination of the appropriate sample size Based on information provided by survey respondents,the range of
can be a major factor in setting an evaluation budget.To address EM&V budgets varies significantly between states from very little to
this,statistical analyses help evaluators determine the sample size about 6%.The 2008 allocation of EM&V budget among different types
needed to ensure that the desired level of precision and confidence of evaluation activities also varied substantially across jurisdictions.Six
for key metrics,or factors that determine key metrics,will be states allocated 75%or more of their EM&V budget for impact evalu-
statistically significant.Additional resources(more samples)might ation,while three states and the Northwest Energy Efficiency Alliance
also be allocated to ensure that"hard-to-reach" portions of the (NEEA)allocated between 50%and 60%of their EM&V budget to
population are included in the sample(reducing non-coverage error) impact evaluations.It appears that many states allocate between 10%
or devoted to follow-up aimed at increasing the number of sample and 20%of their EM&V budget to market research.
members for whom data are obtained (reducing non-response bias).
DOE's Uniform Methods Project includes information on steps that Note that these budget figures should be considered as only rough
can be taken to increase the accuracy of evaluation results with guidance,as they are mostly self-reported,and the definitions that
respect to sampling. are used for what is and is not included in the EM&V budgets varies
significantly between states and program administrators.This is
While it is difficult to generalize,common practice suggests that particularly true when looking at where the line is drawn between
a reasonable spending range for evaluation(impact,process,and activities associated with program implementation and those
market)is 3%to 6%of a portfolio budget.However,this should associated with independent evaluation.
be considered rough guidance,because evaluation needs and the
relative EM&V roles of program administrators and independent 7.6 EVALUATION PRINCIPLES
third-party evaluators(and thus how the budget is categorized
Reliable evaluation of energy efficiency programs requires
between program and evaluation expenses)vary significantly
transparency and independence.This results in high-quality infor-
between different states and different program administrators.In
mation on which business,regulatory,and policy decisions can
general,on a unit-of-saved-energy basis,costs are inversely propor
be made.Therefore,evaluation processes can be defined by the
tional to the magnitude of the savings(i.e.,larger projects have lower
following principles:
per-unit evaluation costs)and are directly proportional to uncertainty
of predicted savings(i.e.,projects with greater uncertainty in the Integral to the portfolio cycle.The evaluation process should be
predicted savings warrant higher EM&V costs). integral to what is typically a cyclic planning-implementation-
evaluation process.Therefore,evaluation planning should be part
7.5.3 State Evaluation Budgets of the program planning process,so that the evaluation effort can
To provide some guidance on overall budgets for evaluation,the not only support program implementation,including the align-
following three tables provide information on the EM&V budgets ment of implementation and evaluation budgets and schedules,
of various states and program administrators.Table 7.2,from the but also provide evaluation results in a timely manner to support
Consortium for Energy Efficiency's(CEE's)2011 annual industry existing and future programs.See Chapter 8 for more information
on the portfolio cycle and the integral role of evaluation.
7-14 www.seeaction.energy.gov December 2012

TABLE 7.2: Electric and Gas EMM Expenditure and Budget Dollars, United States and Canada (millions USD)*
7 mco
2010 EM&V 2011 EM&V BUDGETS TOTAL 2011 ENERGY
EXPENDITURES EFFICIENCY BUDGETS""
United States 58 154 4,239
Canada 11 32 895
Total 69 186 5,134
COUNTRY 2010 EM&V 2011 EM&V BUDGETS TOTAL 2011 ENERGY
EXPENDITURES EFFICIENCY BUDGETS"
United States 9 27 782
Canada 1 Less than 1 78
Total 10 27 860
Source:Wallace,P.;Forster,H.J.(2012).State of the Efficiency Program Industry Budgets,Expenditures,and Impacts 2011.Consortium for Energy Efficiency.
www.ceel.org/files/2011 CEEAnnual Industry Report.pdf.
Notes:*The above table includes only those programs that provided an EM&V dollar figure in response to the survey.Those that provided an estimated percentage of their EM&V
activities from their total energy efficiency funding are not included.**Dollar figures in the Total 2011 Energy Efficiency Budgets column exclude load management because CEE did
not ask for EM&V expenditures and budgets in the load management portion of the survey.
December 2012 www.seeaction.energy.gov 7-15

TABLE 7.3: Ratepayer-Funded Energy Efficiency Budgets and EMU Expenditures for Selected States
REGION
(MILLIONEFFICIENCY (MILLION
BUDGET OF •TAL BUDGET OF •
(MILLION (MILLION
EFFICIENCY EFFICIENCY
BUDGET BUDGET
CA 1014.2 80.2 7.9% 1376.7 79.3 5.8%
I I I I I I
CT 113.6 1.7 1.5% 107.4 1.4 1.3%
FL 124.3 ? NA 138.9 0.1 0.1%
IA 58.9 3 5.1% 90.5 3.2 3.5%
ID 19.7 0.9 4.6% 33.1 0.5 1.5%
IL 41.0 1.3 3.2% 67.4 2.2 3.3%
MA 148.9 5.1 3.4% 208.5 7.8 3.7%
ME 16.8 0.2 1.2% 20.8 0.2 1.0%
MN 136.5 1 0.7% 73.7 1.3 1.8%
N E EA 97.5 1 1.0% 105.2 1.8 1.7%
NY 287.9 7.7 2.7% 421.2 7.6 1.8%
OR 76.8 1.6 2.1% 105.4 2.2 2.1%
PA ? ? NA 8.7 0.1 1.1%
TX 106.4 01 0.9% 101.8 0.2 0.2%
W I 140 2.4 1.7% 162.4 4.8 3.0%
Source:Messenger,M.;Bhorvirkar,R.;Golemboski,B.;Goldman,C.A.;Schiller,S.R.(April2010).Review of Evaluation,Measurement and Verification Approaches Used to Estimate the
Load Impacts and Effectiveness of Energy Efficiency Programs.Lawrence Berkeley National Laboratory.Report LBNL-3277E.http://emp.lbLgov/sites/all/files/Ibnl-3277e.pdf..Source of
data in table as indicated in the original report:Consortium for Energy Efficiency(2008),Consortium for Energy Efficiency(2009),and U.S.Census Bureau.See source report for notes.
7-16 www.seeaction.energy.gov December 2012

TABLE 7.4: EM&V 2008 Funding Levels and Allocation Useful retrospective analyses.The evaluation process is
Among Activities designed to support the policy goals of the energy efficiency
programs being evaluated.As such,evaluations should
STATE/REGION • • develop retrospective estimates of energy savings attributable
OF to a program in a manner that is defensible in proceedings
conducted to ensure that energy efficiency funds are properly
PROCESS/ and effectively spent. Evaluation activities should go beyond
% MARKET RESEARCH) documenting savings to actually improving programs and
providing a basis for future savings estimates.
CA 75/15/10 • Adequate resources. Evaluation budgets and resources should
be adequate to support the evaluation goals and the level of
CT NA quality(certainty)expected in the evaluation results over the
FL NA entire time frame that program impacts need to be assessed.
• Completeness and transparency.Results and calculations
IA 50/30/20 should be coherently and completely compiled.Calculations
should be well documented and transparent,with reported
ID NA levels of uncertainty.The scope of the documentation should
take into account the relevant independent variables that
IL 75/15/10 determine benefits and include a properly defined baseline.
In addition,documentation and reporting should include all
MA 75/15/10 relevant information in a coherent and factual manner that
allows reviewers to judge data quality and results.Among the
ME 30/50/20 key qualities of a good,transparent analysis are the following:
MN NA — Project descriptions indicate the approaches and the vari-
ables used to determine energy savings.
NEEA 60/30/10 — Critical assumptions are stated and documented.
— Documentation is presented in a format that allows the
NY 80/10/10 reviewer to follow a connected path from assumptions to
OR 50/30/20 data collection,data analysis,and results.
— Levels and sources of uncertainty are reported.
PA 50/30/20 Relevance and balance in risk management,uncertainty,and
costs.The data,methods,and assumptions are appropriate
L TX 75/15/10 for the evaluated program.The level of effort expended in the
evaluation process is balanced with respect to the value of the
WI*** 100/0/0 savings,the uncertainty of their magnitude,and the risk of
Source:Messenger,M.;Bharvirkar,R.;Golemboski,8.;Goldman,C.A.;Schiller,S.R. overestimated or underestimated savings levels. Benefits are
(April2010).Review of Evaluation,Measurement and Verification Approaches Used to calculated at a level of uncertainty such that the savings are
Estimate the Load Impacts and Effectiveness of Energy Efficiency Programs.Lawrence neither intentionally overestimated nor underestimated,and
Berkeley National Laboratory.ReportLBNL-3277E.http://emp.lbl.gov/sites/all/files/ the quality of the reported information is sufficient for main-
Ibnl-3277e.pdf.
taining the integrity of the program being evaluated.
*The range depicts answers provided by different respondents from California;the
funding also includes evaluations octivitiesfor codes and standards.NA=not available. • Consistency. Evaluators working with the same data and using
**Annualfunding for the 2009-2013 cycle. the same methods and assumptions will reach the same con-
***Program administrators may conduct some market research during program clusions. In addition,for efficiency programs that are part of
implementation phase;market research is not carried out as part of EM&V activities. broad efforts(e.g., utility resource procurement programs or
emissions cap-and-trade systems),energy and demand savings
December 2012 www.seeaction.energy.gov 7-17

and avoided emissions calculated from one program area are website at www.eval.org,these principles are summarized here:
as valid as those generated from any other actions,whether . Systematic inquiry.Evaluators conduct systematic,data-based
demand-side or supply-side.This allows for comparison of the inquiries.
range of energy resources,including energy efficiency.
• Competence. Evaluators provide competent performance
With counterfactual baselines,uncertainty is inherent and savings to stakeholders.
estimates are prone to a certain degree of subjectivity.Because of • Integrity/honesty. Evaluators display honesty and integrity in
this subjectivity,and possibly a lack of relevant information,some their own behavior and attempt to ensure the honesty and
believe that"conservativeness"should be added to the list of integrity of the entire evaluation process.
principles for the purpose of counteracting what is seen by some as a . Respect for people. Evaluators respect the security,dignity,
natural tendency toward savings inflation.There are many real-world and self-worth of respondents,program participants,clients,
incentives for people to over-report savings,and fewer incen- and other evaluation stakeholders.
tives working the other way.This subjective bias may be difficult Responsibilities for general and public welfare. Evaluators
to keep in check without an explicit directive to be conservative. articulate and take into account the diversity of general and
However,many in the evaluation field believe that credibility, not public interests and values that may be related to the evaluation.
conservativeness,is the desired characteristic,and that underesti-
mates can be just as biased and damaging as overestimates.Thus, 7.7 USING IMPACT EVALUATIONS FOR
the correct guidance is to develop the"most likely"result and not PROGRAM FEEDBACK
one that is biased to be conservative or aggressive.
Impact evaluation results are used to make informed decisions
Beyond the characteristics of the evaluation itself,evaluations can on program improvements and future program designs and offer-
only be effective if those conducting the evaluations perform their ings throughout the efficiency portfolio implementation cycle.
tasks fully and completely,and are free of bias without a stake in the The implementation cycle is one in which programs are designed,
outcome,with respect to the performance of the programs under then implemented,and then evaluated.Using the results of the
consideration.Related to the characteristics of the evaluation itself, evaluation,programs are reexamined for design changes that may
the credibility of evaluators is essential for providing credible findings be needed.This cycle provides a continuing process of program
on the results from the program and for providing recommendations improvement,so that the programs match available market opportu-
for program refinement and investment decisions. nities and continually improve their cost-effectiveness over time.The
impact evaluation planning process is discussed more in Chapter 8.
The relationship between the evaluator and the implementers
and/or administrators—whose work is being evaluated—needs to Impact evaluations tend to be a retrospective process for
be cooperative.This allows for information sharing,access to project determining how a program performed over a specific period of time
sites,and for the results of the evaluator to be considered valid by (e.g.,month,season,year); nonetheless,evaluations that produce
the implementers and administrators and thus considered as useful results while the program is operating can be very useful.When
input for program improvement. However,there will always be some possible,evaluations should be done within a program cycle so that
stress in the relationship as(1)the evaluator cannot allow itself to feedback is timely and systematic,benefits the existing program,and
be unduly influenced by the implementer/administrator,or for that informs the design of future programs and their evaluation.
matter,whoever hires the evaluator,including an entity such as a
state regulator;and(2)the administrator/implementer will have a For planning a future program,historical evaluation results can help
sense that their work is being judged by the evaluator,because the with program design.However,for estimating how a program will
evaluator may very well have a significant say in the compensation or perform,potential studies and feasibility studies are the typical
penalties applied to the implementers and administrators. analyses performed. Both of these studies look at what levels
of savings are possible from technical,economic,and market-
Thus,evaluation ethics are a critical foundation for the activities acceptance perspectives. Potential studies are typically conducted on
described in this guide.The American Evaluation Association(AEA) a market-sector(e.g.,residential,commercial,industrial)basis and
has a set of guiding ethical principles for evaluators. Located on AEA's feasibility studies tend to be focused on specific customers that may
be involved in a particular program.
7-18 www.seeaction.energy.gov December 2012

RESOURCE PLANNING: NORTHWEST POWER AND CONSERVATION COUNCIL
Since 1980,the Northwest Power and Conservation Council has inputs for its assessment of the remaining potential for
been charged with preparing regional integrated resource plans energy efficiency in the region. For example,the forecast-
for the Pacific Northwest states of Idaho,Oregon,and Washington, ing model uses the sales-weighted average efficiency and
as well as the western portion of Montana.The council plans energy use of new clothes washers derived from market
consist of forecasts of electric loads and a least-cost/risk-resource assessments and program impact evaluations to compute
portfolio to meet those regional electric demands over a 20-year the regional energy demand created by this appliance.The
time frame.All of the council's plans have relied heavily on energy baseline use and efficiency for clothes washers in the coun-
efficiency.Therefore,the council's resource-planning process is cil's energy efficiency potentials assessment is the same as
critically dependent on data derived from market characterization that used in the load forecast.Thus,for resource-planning
research and impact evaluation results.Results of these stud- purposes,the forecast of future demand and the potential
ies support load forecast model calibration and assessments of for reducing that demand start with the same value.One
energy efficiency potential in the following ways: implication of this approach for impact evaluation is that
• The council uses an end-use econometric load forecast the use in the forecast and potentials assessment of a
model. In simple terms,this model calculates future "market average efficiency"(even if it is more efficient
electricity sales by multiplying the energy use of a given than local code or federal standards)reflects the efficiency
device(e.g.,clothes washer, refrigerator)by the expected choices that are occurring absent future program effects.
number of those devices in the region. Market characteriza- As a result,at least in theory,the only difference between
tion studies,such as residential and commercial appliance gross and net savings is potential spillover impacts.
and equipment surveys,are used to estimate the saturation The council's assessment of energy efficiency potential and
of each end-use device included in the forecast model. cost(which is also sometimes based on program evaluation
In addition,these surveys are also used to establish the data)is used for resource planning. Because the council
baseline characteristics of the existing stock,such as the treats energy efficiency as a resource,it competes directly
average efficiency of equipment and the lighting power against generating resources in the agency's resource plan-
density of residences and commercial buildings. In addition ning model.That is,rather than reduce the load forecast
to"savings,"impact evaluation results,especially those that based on estimates of achievable potential,the amount of
provide estimates of both pre-and post-measure adoption energy efficiency to be acquired is based on testing its cost
energy use,are used to calibrate the forecasting model to and availability against supply-side resources.Thus,energy
actual observed use. savings from impact evaluations are used directly to derive
• The council forecast model's estimates of end-use energy the future need for supply-side resources.
consumption and efficiency are used as the"baseline" Provided by Tom Eckman of the Northwest Power and Conservation Council
7.8 EVALUATION FOR DEMAND FORECASTING Total ratepayer-funded energy efficiency program spending
AND ENERGY RESOURCE PLANNING in the United States(80%of which is targeted to electric end
Efficiency is an important energy resource that is being recognized as uses)is projected to rise from$5.4 billion in 2010 to$7.0
such in state and regional energy resource forecasting and planning billion in the low scenario by 2025,$10.8 billion in the medium
efforts.77 This is driven in part by at least 24 states having enacted scenario,and$16.8 billion in the high scenario for both electric
long-term(three-plus ears specific energy-savings tar ets,which and gas efficiency programs.
g ( p years) pg
can take the form of a statewide energy efficiency resource standard While the West and Northeast combine for more than 70%of
(EERS).78These EERS targets include goals calling for efficiency to efficiency program spending in 2010, by 2025,it is projected
reduce electricity consumption on the order of 1%to 3%per year that 50%of the spending will be in the South and Midwest.
and/or meet 30%to 100%of increases in demand for electricity. At a national level,the analysis suggests that savings from
Looking to the future,a 2012 Lawrence Berkeley National Laboratory energy efficiency programs in the medium scenario have the
report found the following:79 potential to offset up to 80%of the projected growth in U.S.
electric load in 2020.
December 2012 www.seeaction.energy.gov 7-19

An effect of these actions is that efficiency is,and will increasingly Impact studies that consider savings attribution can also be used to
become,a resource that has to be accounted for in state and transparently indicate the sources of changes in demand forecasts
regional energy-demand forecasts and related capacity,transmission, (whether from voluntary efficiency programs,codes and standards,
and distribution resource planning efforts. economic conditions,changes in social norms,or other factors).
Cost-effectiveness analyses can be used in integrated resource
Market and impact evaluations play an important role in supporting planning efforts for comparing efficiency with other energy resources
these resource planning and forecasting activities. Market potential such as conventional and renewable power plants,as well as
studies can be used to project savings that will be available from location-specific transmission and distribution requirements.
future efficiency efforts and impact evaluations,and market effects
evaluations can be used to calibrate demand forecast models. For evaluations to support demand forecast and resource plan-
ning efforts,it is important for the evaluation planning efforts to
RESOURCE PLANNING: consider the metrics and data requirements,as well as time frames
ENERGY TRUST OF OREGON of the people conducting the resource forecasting and planning
efforts.These people include forecasters working in utilities,state
Energy Trust of Oregon(ETO)impact evaluations,process energy offices,and/or utility commissions;independent system
evaluations,and estimates of market effects are used directly operators;regional transmission organizations;and regional energy
for integration of efficiency into utility resource planning. and national energy planning organizations.The terminology used
ETO provides energy efficiency savings under contract,and in demand forecasting and resource planning,baseline scenarios,
develops efficiency supply curves and deployment scenarios historic and future planning time frames—as well as the format and
in Oregon for PacifiCorp,Portland General Electric,Northwest definition and granularity of data requirements(e.g.,coincident peak
Natural,and Cascade Natural Gas. ETO also provides the versus average peak demand reductions and source versus end-use
same services for Northwest Natural in their Washington energy savings)—can be different for the resource planner and
territory.This occurs through the following steps: forecaster than they are for a regulator of a retail energy provider.
1. ETO develops supply curves—estimates of the cost and 7,9 DETERMINING NON-ENERGY BENEFITS
savings available from various conservation measures
Virtually all energy efficiency programs have objectives associ-
and sectors. For each integrated resource planning
ated with reducing energy use and costs.This guide focuses on
(IRP)cycle,estimates of costs are updated as needed
documentation of these energy-and demand-related benefits and
based on ETO program data.The estimates of savings
the associated avoided emissions. However,there is a wide range
are adjusted based on the most recent available
of other benefits that come from energy efficiency activities;for
evaluations.
2. ETO develops deployment scenarios that provide a consumers,these non-energy benefits(NEBs)may actually drive
recommended rate of acquisition over 20 years.Impact
their interest in efficiency investments. Historically,these NEBs have
and process evaluations inform that rate to the extent
been,mostly,just subjectively noted as"other benefits,"with little
that they show where and how ETO's programs are quantification or documentation of their value. Not putting a value
reaching customers.These scenarios show"net"savings to these NEBs,assuming they are positive,can result in negative
after market effects,based on trends from recent
bias in energy efficiency program investment decisions and less than
evaluations.
fully effective program participation,designs,and marketing.The
3. Utilities integrate efficiency into their linear planning later would happen because the program implementers might not
models for resource selection(as part of the integrated
focus on the same benefits that participants focus on when they are
planning process)in a variety of ways.The common making decisions about implementing efficiency projects.
factor is that the measures that are less costly than NEBs can be categorized as those accruing to utilities(energy
generation in the long term are selected.This is affected providers),society as a whole,and to individual participants.80 Some
by the savings estimates. research indicates that the value of benefits to society as a whole
4. ETO funding to pursue future conservation is determined and individual participants make up the bulk of the value of NEBs.
based on goals coming out of this IRP process.Success is
then judged by evaluation estimates of net savings.
Provided by Fred Gordon of Energy Trust of Oregon
7-20 www.seeaction.energy.gov December 2012

NEW YORK STATE ENERGY RESEARCH AND DEVELOPMENT
AUTHORITY • OF •
New York State Energy Research and Development Authority All of these approaches have benefits and drawbacks.The industry
(NYSERDA)considers that non-energy benefits can be evaluated standard has been CV and DQ approaches.However,NYSERDA has
through a range of approaches: pioneered the joint use of DQ and CA survey methods on its prior
• Contingent valuation(CV)survey techniques directly ask New York Energy$martSM Program and is mainly employing DE
respondents'willingness to pay for a particular good. approaches currently.
• Direct query(DQ)approaches ask respondents to value DQ approaches have somewhat consistently produced NEB values
NEBs relative to a given parameter,such as the energy around 50%of the value of the program-induced energy savings.
savings achieved on their project.To assist respondents, CA approaches have resulted in similar results for some programs,
these surveys often use a scale or provide the dollar value but widely varied results for other programs ranging from 4%to
of the energy savings. 340%of the value of the program energy savings.DE approaches,
• Conjoint analysis(CA)survey techniques provide which only measure the more quantifiable non-energy impacts,
respondents with descriptions of different scenarios can result in relatively lower values as compared to DQ and
or levels of NEBs,asking them to either rank or choose CA approaches.NYSERDA's recent commercial/industrial NEBs
between the different options presented. Econometric research that found DE-based NEBs of$3 to$4 per natural gas
techniques are then applied to calculate the"utility"or MMBtu saved and$5 per MWh of electricity saved.The value of
relative value of each attribute. these NEBs to participants ranged from 6%to 39%of the retail
• Direct estimates(DE)of non-energy impacts can be made value of their energy savings at typical New York utility rates.
in conjunction with impact evaluation site visits,and Source:JenniferMeissner,Program Manager,Evaluation,NYSERDA.2012.
generally focus on operations(e.g.,water savings),materials, References include:Energy&Resources Solutions,Inc.(ERS);Left Fork Energy;West
Hill Energy&Computing;Megdal&Associates,LLC.;Opinion Dynamics Corporation.
and labor hours due to reduced maintenance requirements. (June 2010).Report on Impact Assessments of Two NYSERDA Natural Gas Efficiency
These estimates are based on equipment specifications, Programs:Con Edison Natural Gas Efficiency Program&National Grid Low Income
operating parameters,and other customer-supplied Natural Gas Efficiency Program.Preparedfor the New York State Energy Research
and Development Authority(NYSERDA).http://documents.dps.ny.gov/public/
information,and they are monetized at the project level. Common/ViewDoc.aspx?DocRefld=(CFCD7DE1-8868-46CC-8648-83ACD072A4FD).
The following are examples of utility NEBs:81 these effects on marginal peak costs are not always included in
• Avoided transmission and distribution capital and operating benefit-cost analyses)
costs(particularly in areas with high energy use, high demand • Reduced air emissions,water use,water pollution,and the
growth,and/or constrained distribution systems) related emission trading values and/or health/hazard benefits
• Reduced line losses,voltage support(reliability),and power • National security improvements through reduced reliance on
quality improvements certain fuels.
• Customer bill collection and service-related savings such as
Examples of efficiency program participant NEBs include the following:
avoiding notices,shutoffs/reconnects,and carrying costs on
arrearages. Indoor air quality improvements,improved comfort(e.g.,quality
of light,less noise,fewer drafts,better building temperature con-
The following are examples of societal NEBs: trol),higher productivity and lower rates of absenteeism through
• Economic development—for example,job development,both better-performing energy using systems(e.g.,ventilation,building
direct and indirect shell,lighting)
• Energy price effects,such as stability, lower peak costs,and Reduced equipment operations and maintenance(O&M)costs
downward pressure on wholesale energy prices(although because of more efficient,robust systems(although more
these price effects can also be considered"energy"benefits, complex systems could require more maintenance).
December 2012 www.seeaction.energy.gov 7-21

JOB OF • ' POLICIES
The following information is from a report presenting the results of the California Workforce Education and Training Needs Assessment
for Energy Efficiency,Demand Response,and Distributed Generation.The assessment was conducted throughout calendar year 2010.
Findings of the report are summarized in the table below and also include the following:
• By 2020,energy efficiency policies will result in about$11.2 billion of public and private investment,resulting in 211,000 direct,
indirect,and induced jobs.
• Two-thirds of the direct jobs are in traditional construction trades,one-sixth are in building professional occupations.
• There are many more jobs are in traditional trades occupations than in new or specialized occupations.
• There are many more incumbent workers than new workers,which indicates the need for training in that there are relatively
few slots for job seekers per year compared to the stock of current workers who may need skills upgrading.
• There are many more displaced construction workers than jobs,indicating that there are more unemployed,experienced con-
struction workers due to the recession than there are job openings created by public/ratepayer investment in energy efficiency.
EQUIVALENT •
INDUSTRY/OCCUPATION GROUP
Residential Building Construction 5,072 7,104
Nonresidential Building Construction 5,342 6,924
Electrical Contractors 319 1,649
LPlumbing,Heating,and AC Contractors 4,859 9,407
Drywall and Insulation Contractors 451 840
Manufacturing 51 574
Advertising and Related Services 956 1,794
LEngineering Services,Architectural Services,etc. 2,118 4,026
Management, Public Administration 1,231 2,449
Office Administrative Services 2,021 3,958
LAll Other Industries 108 212
Total 22,528 38,937
The job increases are compared to the base year of 2009.They are in job person years(i.e.,the number of full-time equivalent jobs
for one year).They are not permanent jobs in that they depend on the annual investment of energy efficiency dollars.
Source:Zobin,C.(2011).California Workforce Education and Training Needs Assessment.Center for Labor Research&Education,Donald Vial Center on Employment in the
Green Economy.www.irle.berkeleyedu/vial.
7-22 www.seeaction.energy.gov December 2012

• Water and wastewater savings 82 work that a program is targeting(e.g.,construction,engineering,
• Positive personal perceptions(e.g.,"green,"environmental con- architecture).
sciousness)and for commercial businesses and public entities, • Indirect. Indirect jobs are jobs in firms supplying goods and
improved public perceptions and the ability to market products services to energy efficiency firms(e.g., manufacturing,
and tenant leases accounting).
• Avoided capital cost for equipment or building component • Induced.Induced jobs are those created by the demand
replacements whose capital costs can be paid from savings. generated by wage and business income from energy efficiency
investments and by energy bill savings(e.g.,jobs induced by
While most NEBs are considered to be positive,they can also be NEBs,such as grocery store or apparel).
negative.83 Examples of negative NEBs include aesthetic concerns
associated with fluorescent lamps,"hassle"factor associated with The methods for documenting NEBs tend to fall into one or more of
implementing projects,and increased maintenance costs due to the following three categories:
unfamiliarity with new energy efficient equipment or the need Measurement of benefits.These methods are used with
to operate more sophisticated equipment(e.g.,building control benefits that can be directed, measured,or calculated,such as
systems)on a continuous basis. water savings.As with energy savings,a variety of"measure-
As noted above,a common impact evaluation approach to NEBs is to ment"approaches are possible,including the use of control
list them as possible or probable benefits and not to quantify them. groups and M&V. In some cases,the measurements can be
This is typically because of program administrators'(and sometimes made of secondary metrics that are indicators. For example,
evaluators')unfamiliarity with methods for quantifying these ben- comfort can be defined via monitoring of indoor temperatures
efits,the cost of quantifying them,and the sense that the majority of and humidity to document whether participants' homes,
economic benefits are associated with saved energy costs.However, compared with those of non-participants,are within defined
the methods for documenting NEBs are improving and expertise in comfort conditions.85
this area is increasing.And,perhaps most important,it is becom- • Modeling.These methods include macroeconomic modeling
ing increasingly clear that NEBs can have very high value for those and analysis tools that look at broader societal impacts such as
making decisions about efficiency projects and to society as a whole. job growth or modeled estimates of emissions impacts.
• Surveys.These are used for documenting many different types
In fact,for some programs,it appears that these participant NEBs of NEBs.There is a wide range of surveying approaches to
can exceed the energy-related benefits.84 For example,in a com- determine these benefits.These include willingness to pay and
mercial office building where the cost of the employee salary and willingness to accept contingent valuation(CV)studies,compara-
benefits per square foot dwarfs the cost of energy per square foot, tive or relative valuations,and other revealed preference and
an increase in employee productivity of a few percent may be signifi- stated preference approaches.Surveys are used specifically for
cantly more valuable than decreasing energy costs by 30%. Including determining relatively subjective program participant benefits
these non-energy benefits in evaluations can be quite valuable,given when quantification is difficult and/or expensive.86 However,
that energy efficiency programs increasingly are emphasizing these surveys can be used for almost all benefit types where partici-
types of participant non-energy benefits when marketing energy pants and non-participants can be asked to provide data(e.g.,
efficiency programs to customers.Those factors arguably play an how many people they hired for determining job impacts,
important role in persuading customers to make the significant whether they believe their indoor air quality is better,if there
investments necessary to achieve comprehensive energy savings in are distribution projects that were delayed).
a home or business.
Currently,several states are including NEBs in their evaluations,but
In terms of societal NEBs,one benefit that has generated a great deal not many. In particular for cost-effectiveness analyses,the ACEEE
of interest is job creation.Jobs are created as a result of efficiency 2012 review of evaluation practices87 indicated the following:
programs in three categories(see the California Efficiency Policy
....while 36 states(including all the states with TRC[total
sidebar for examples of results ofjobs analyses):
resource cost]as their primary[cost-effectiveness]test)treated
• Direct. Direct jobs are in firms that are actually receiving the "participant costs"for the energy efficiency measures as a
efficiency program dollars and doing the energy efficiency cost,only 12 states treated any type of participant"non-energy
December 2012 www.seeaction.energy.gov 7-23

benefits'as a benefit.... most of those"non-energy"participant Obstacles to the widespread adoption of behavior-based programs
benefits were confined to"water and other fuel savings."Only include issues relating to whether these programs can be evaluated
2 states quantified a benefit for"participant 0&M savings" in a rigorous way,the savings persist,and the evaluated results
and none quantified any benefits for things like"comfort," shown for one program can be applied to another program.Another
"heath,""safety,"or"improved productivity"in their primary SEE Action report88 specifically addresses behavior-based programs
benefit-cost test. and prepares recommendations with respect to these issues.The
following lists the primary recommendations from that report:
Given the potential significant value of NEBs,it is expected that For program evaluation design,the use of randomized con-
more jurisdictions will analyze NEBs and that more cost-effectiveness trolled trials(RCT)is recommended.These will result in robust,
analyses will take them into consideration,such as in the societal unbiased estimates of program energy savings. If this is not
cost test. feasible,it is suggested that"quasi-experimental"approaches
7.10 IMPACT EVALUATION FOR UNIQUE be used.
ENERGY EFFICIENCY PROGRAM TYPES For a level of precision that is considered acceptable in
7.10.1 Residential Behavior-Based Programs behavioral sciences research,it is recommended that a null
hypothesis(e.g.,a required threshold such as the percent sav-
Encouraging people and organizations to do things differently ings needed for the benefits of the program to be considered
is a tricky proposition. Energy efficiency is not just about new cost-effective)should be established.The program savings esti-
technologies;it is about new behaviors and better decisions.For mate should be considered acceptable(i.e.,the null hypothesis
more than three decades,program administrators have offered should be rejected)if the estimate is statistically significant at
consumers energy efficiency programs that have used strategies the 5%level or lower.
such as subsidies,rebates,or other financial incentives to motivate
• In order to avoid potential evaluator conflicts of interest,it is
consumers to install technologies and high-efficiency measures. In
recommended that results are reported to all interested parties
many respects,these programs have focused on affecting behaviors
by giving people incentives to undertake efficiency actions.
and that an independent third-party evaluator transparently
defines and implements the following:
However,in the last several years,there has been substantial interest — Analysis and evaluation of program impacts
in broadening energy efficiency program portfolios(particularly the — Assignment of facilities(e.g.,households)to treatment and
residential programs)to include behavior-based programs that use control groups(whether randomly assigned or matched)
strategies intended to affect how consumers use energy in order to — Selection of raw utility data to use in the analysis
achieve energy and/or demand savings.These programs typically — Identification and treatment of missing values and outliers
include outreach,education,competition,rewards,benchmarking, — Normalization of billing cycle days
and/or feedback elements. In some cases,this new generation of — Identification and treatment of households that close
programs takes advantage of technological advances in Internet and their accounts.
wireless communication to find innovative ways to both capture
energy data at a higher temporal and spatial resolution than ever For the analyses of savings, it is recommended to use a panel
before and to communicate the energy data to households in creative data model89 that compares the change in energy use for the
new ways that leverage social science-based motivational techniques. treatment group to the change in energy use for the control
group,especially if the evaluation design is quasi-experimental.
These programs are unique in that they may rely on changes to
consumers'habitual behaviors(e.g.,turning off lights)or"one-time" With respect to external validity of behavior-based program evalua-
behaviors(e.g.,changing thermostat settings). In addition,these lion results,it is possible in theory that a predictive model could be
may target purchasing behaviors e. created that allows program estimates to be extrapolated to future
programs y g p g ( g.,purchases of energy
efficient products or services),often in combination with other years and new populations without actually measuring the savings
programs such as rebate programs or direct install programs.These estimates in those years. It is also possible that behavior-based pro-
grams could move to a deemed or modeled savings approach over
programs are also distinguished by normally being evaluated using
large-scale data analysis approaches involving randomized controlled time. However,it is generally believed that the industry is not yet
trials or quasi-experimental methods versus deemed savings or at this point,due to the small number of behavior-based programs
that have been evaluated using rigorous approaches.
M&V approaches.
7-24 www.seeaction.energy.gov December 2012

For more details and recommendations on evaluating behavior- this type of evaluation are surveys and focus groups.The following
based programs,please see the 2012 SEE Action report referenced are examples of information topics that may be collected as part of
above.Another source of information on behavior-based programs surveys and focus groups(paraphrased from the previously cited
and their evaluation are the proceedings of the Behavior, Energy,and California Energy Efficiency Evaluation Protocols):
Climate Change Conferences(http://beccconference.org/). Information and education program evaluation topics:
7.10.2 Education and Training (E&T) Programs — Number and percent of customers reached or made aware
Education and training(E&T)programs are seen as very important — Number and percent of customers reached who take
strategies for expanding energy efficiency's reach as a sustained, recommended actions
long-term resource. Education and training programs may be — Number and type of actions taken as a result of the program
targeted to either end-use customers or other market actors(e.g., — Changes in awareness or knowledge by topic or subject area,
trade allies)whose activities influence the energy-related choices of by type of customer targeted
end-use customers.These programs can include advertising,public — Customer perception of the value of the information and/or
service announcements,education efforts,training activities(e.g., education received
for contractors,building operators,and designers),outreach efforts, — Elapsed time between information exposure and action(s)
demonstration projects,and other information or communication- taken by type of customer targeted
based efforts. — Attribution of cause for actions taken when multiple causes
Typically,E&T programs have one or more of the following general may be associated with the actions taken
goals or desired outcomes: — Influence of the program on dealers,contractors,and
trade allies
• Educate energy consumers regarding ways to increase the
— Effects of the program on manufacturers and distributors.
energy efficiency of their facilities and activities,and thus
convince them to take actions that help them manage their Training program evaluation topics:
consumption or adopt more energy-efficient practices. — Pre-program level of knowledge to compare with post-
• Inform energy consumers and/or other market actors about program levels
program participation opportunities in order to increase — The specific knowledge gained through the program
enrollment in these programs. — The relevance and usefulness of the training as it relates
• Inform energy consumers and/or other market actors about to the participants'specific needs and opportunities to use
energy issues,behaviors,or products in an effort to transform the information
the normal operations of the market. — Future opportunities and plans for incorporating the
• Train contractors,engineers,architects,and building operators knowledge gained into actions or behaviors that provide
on skills, best practices,tools,and other issues related to energy impacts
energy efficiency.An example program is the Building Operator — Whether participants would recommend the training to a
Certification Program(www.theboc.info). friend or colleague
— Participant recommendations for improving the program.
Almost every energy efficiency program provides some level of
educational and/or informational content. However,education-or Note that programs with large training efforts,or programs designed
training-specific programs are typically designed to achieve energy or solely for training,should have evaluation designs that are mindful of
demand savings indirectly.Thus,while they are important strategies, the rich literature and methods on evaluating training programs that
these programs only indirectly result in energy and demand savings are available from the larger(education)evaluation community.
and therefore represent unique impact evaluation challenges. 7.10.3 Market Transformation (MT) Programs
For E&T programs,evaluations usually focus on documenting the There are many definitions of market transformation(MT),although
degree to which the programs are achieving their desired outcomes it is often considered the ultimate goal of publicly and consumer-
within the markets targeted by the program,which is educating and funded efficiency programs.In this guide,the definition of market
training people on energy efficiency.The primary mechanisms for transformation is:"a reduction in market barriers resulting from a
market intervention,as evidenced by a set of market effects(or
December 2012 www.seeaction.energy.gov 7-25

perhaps more specifically a set of market progress indicators)that interventions.This role is equally important for resource acquisition
are likely to last after the intervention has been withdrawn,reduced, and MT interventions,but arguably more complex for MT programs,
or changed."MT denotes a permanent,or at least long-term,change because the interest is in long-term changes in the market versus
in the operation of the market for energy efficiency products and more immediate and direct energy savings for resource acquisition
services.As such,their"end-point"can be considered to have programs.
occurred if the subject energy efficiency measure(s)or practices are
either simply part of common practice(i.e.,all consumers,designers, As a final note for MT program evaluations,and as discussed briefly
builders,and operators just"do it")or required per a code in Appendix B and the sidebar"Theory-Based Evaluation:A Guiding
or standard. Principle for Market Transformation Evaluation,"a program logic
model and a program theory are common components of MT programs
Market transformation programs attempt to reduce market barriers and their evaluations.A program logic model is a visual representa-
through market interventions.During the 1990s,the focus of tion of the program's theory that illustrates a set of interrelated
many energy efficiency efforts shifted from resource acquisition program activities that combine to produce a variety of outputs that
to market transformation.Subsequently,there was a shift back to lead to key(in these programs)MT outcomes.Thus, MT program
more resource acquisition-focused programs that did not necessarily evaluations should entail collecting information that can be used to
include market intervention components,or at least did not include refine the underlying program theory(see sidebar).
components defined as such. However,current best practices have all
publicly or energy consumer-funded programs having at least some 7.10.4 Codes and Standards Programs
MT elements,in that they involve changing how energy efficiency As mentioned in the previous section,from a market transformation
activities take place in the marketplace.Thus,MT and other program perspective,the end goal of energy efficiency programs is to have
types are now often implemented in a complementary manner. effective energy efficiency actions become either common practice
or established in energy codes or standards(C&S). Building energy
MT evaluation tends to be a combination of impact,process,and codes and standards set minimum requirements for energy efficient
market effect evaluation,and can also include cost-effectiveness design and construction for new and renovated buildings.Appliance
evaluations.However,given that the ultimate aim of MT programs (equipment)standards set minimum requirements for energy
is to increase the adoption of energy-efficient technologies and consumption of appliances and equipment.States and local govern-
practices,MT evaluation usually focuses first on energy efficiency ments generally establish building energy codes applicable in their
adoption rates by market actors and second on the directly associated jurisdictions and both states(approximately 15)and the federal
energy and demand savings. Market actors that influence end-use government establish equipment standards.90
consumer choices include installation and repair contractors,
retailer staffs,architects,design engineers,equipment distributors, Local governments,states,and the federal government also establish
manufacturers,and of course the consumers themselves.Also,MT energy efficiency programs to support the development of,imple-
programs are dynamic,and thus the nature of market effects can mentation of,and compliance with C&S.These programs include
be expected to vary over time.Thus,market effects evaluations, efforts such as emerging technology programs,compliance-enhance-
conducted at several points in time,are the primary evaluation ment programs,and stretch(or reach)goal programs,as well as
activity usually associated with MT programs.See Appendix B for a training on C&S for building code officials,builders,contractors,
description of these types of evaluations. and designers.States are interested in C&S and efforts/programs to
support them such as increased code enforcement,because these
Evaluation of MT interventions also needs to focus on the support energy savings goals as well as air pollution reduction and
mechanisms through which changes in adoptions and energy use GHG mitigation goals.
are ultimately induced.This means that considerable attention must
be focused on indicators of market effects through market tracking. When determining energy(and demand)savings from C&S,there
For example,an MT evaluation might first report changes in sales are two aspects to the project scenarios:C&S requirements and
patterns and volumes for particular energy efficiency products as an C&S enforcement/compliance(noting that enforcement is not the
indication of program progress in meeting program goals. same as compliance,compliance being the actual goal that results in
energy savings).Whether a C&S is actually being complied with can
Evaluation also plays an important role in providing the kind be determined through surveys of practices in the field.This can be
of feedback that can be used to refine the design of market done through review of permits. However,that can be quite difficult,
7-26 www.seeaction.energy.gov December 2012

given the lack of digitized and organized permits in many jurisdictions THEORY-BASED
and because permits are often not"pulled"for many projects such as GUIDING PRINCIPLE - MARKET
small HVAC and lighting retrofits.Also,"percent compliance"is not TRANSFORMATION EVALUATION
a very definable concept,given that compliance is usually not a"yes
or no"issue,as partial compliance is often the case.Thus,energy Theory-based evaluation(TBE),an evaluation approach that
performance metrics,such as energy use per square foot for the has been widely used in the evaluation of social programs in
buildings of interest(over an extended period of time),can be a much other fields,has seen increased use in the energy-efficiency
better targeted metric for gauging whether C&S are being complied industry. It involves a relatively detailed and articulated
with and to see if there are any energy savings taking place. program theory,established up front,that specifies the
sequence of events a program is intended to cause,along
The approaches to determining energy savings associated with C&S with the precise causal mechanisms leading to these events.
efforts can involve deemed values,M&V approaches,and large- Evaluation then focuses on testing the congruence of
scale consumption data analyses approaches.However,the usual observed events with the overall program theory.
approach is a combination of calculations and field data collection
that involves the following: A TBE can be considered a process of determining whether a
• Establishing an efficiency baseline for each energy efficiency program theory is correct or not(i.e.,testing a hypothesis).For
measure based on analyses of common practices or naturally example,with an incentive program,the theory is that paying
occurring market adoption rate(NOMAD) a certain level of incentives will result in a certain level of
• Defining unit energy savings per measure per year based measure installation,resulting in energy and demand savings.
on C&S compliant measures'versus baseline measures' Having well-defined program theories helps focus an evaluation
efficiency/performance objective on assessing the validity of those theories,primarily
• Defining a market volume baseline for how many units per to see whether a program concept is successful and should be
year are installed in the marketplace expanded and/or repeated.
• Calculating an energy and demand savings value TBE is particularly well adapted to evaluating the effectiveness
• Adjusting for compliance rates(which for a specific code or of market transformation initiatives.This is largely because
standard may be the compliance rate found in the market or, market transformation tends to take a relatively long time
for a program that supports C&S,may be the improved rate to occur,involves a relatively large number of causal steps
as a result of the program,compared against a baseline and mechanisms,and encompasses changing the behavior
compliance rate). of multiple categories of market actors—all of which makes
it particularly important to focus on specifying and testing a
Figure 7.2 illustrates the components of C&S program evaluation. detailed and articulated program theory.
In terms of programs intended to improve C&S compliance,the
ultimate measure for such efforts is of course better compliance.
However,intermediate metrics may also be important to analyze, • Are there changes in attitudes in code enforcement offices
particularly given the long-term perspective associated with a C&S and/or perceptions of improved enforcement efforts from
effort.The following questions can be raised to discover more about code officials?
these intermediate metrics:
• Are training programs being offered?What are the results When attempting to attribute achieved savings from C&S to a
of pre-and post-test knowledge changes as a result of the specific program or group of programs,it is important to review
training?How many trained builders and building officials are the entire C&S process.This can include interviewing stakeholders,
there as a result of the program?
assessing their effectiveness,and documenting the program contribu-
tions. In these attribution efforts,evaluators often use a preponder-
• Have there been reach/stretch C&S established,and what is ance-of-evidence approach to forming a judgment,recognizing that
the participation level as a result of the program? attribution is elusive when so many parties are involved.
December 2012 www.seeaction.energy.gov 7-27

7.10.5 Demand Response (DR) Programs In most cases,DR programs incorporate and rely on electric
Demand response(DR)is the reduction of a consumer's energy consumption data for the participants being collected and recorded on
use at times of peak use in order to help system reliability,reflect at least a 15-minute basis.Therefore,demand is continuously moni-
market conditions and pricing,or support infrastructure optimization tored,including during the DR events.The savings are the difference
or deferral of additional infrastructure.DR programs may include between the electricity consumption(demand)during an event and
contractually obligated or voluntary curtailment,direct load control, the baseline.
and pricing strategies.As noted in the demand savings section Thus,the pertinent question about DR program evaluations is a
above, DR programs reduce a utility customer's electricity demand in common one:"What is the baseline?" For DR programs,the baseline
response to dispatch instructions or price signals sent to the program is defined as part of the program design,and because participants
participants—a"call"for reductions.Thus,the DR peak demand are typically paid on performance(i.e.,how much they reduce their
savings is determined for when there is a"call"for program partici- demand),the baseline is also defined in the DR participant contract.91
pants to reduce their energy consumption rate.When this call occurs Typical baselines are the demand for energy the hour before the
is referred to as the DR"event." event,the hour after an event,the average of hour before and after,
Demand response includes an array of programs relying on various or the demand for a day similar to the one when the event occurred
incentive mechanisms.Some DR programs simply involve a contract (e.g.,if the event was on a summer Tuesday afternoon,the baseline
with a utility customer to reduce their load when asked(or told)to might be the average of all Tuesday afternoons that summer). In
do so;this is historically called interruptible electricity contracts or some cases,the baseline might be determined with a control group,
rates. Other DR programs include residential air-conditioner cycling, such as if a large-scale consumption data evaluation approach is
where demand savings result from a load control device that actively used and the metric of interest is demand reduction for all of the
reduces electricity demand when activated by a command from a participants,not just individual participants(this might be the case
central control center. Dynamic pricing is another form of DR,where for evaluation of a rate schedule used to induce DR in consumers).
a price signal is sent to consumers,with the expectation that they will 7.1 U.6 Greenhouse Gas (GHG) Mitigation Programs
respond to a higher price by reducing consumption(and therefore In Chapter 6,the documenting of avoided emissions—including
demand).Most DR programs require active and continued partici- greenhouse gases(GHG)—that result from energy efficiency pro-
pation because it is assumed that once a participant is no longer grams is described,including issues associated with documenting
enrolled in the program,electric loads revert back to normal patterns. avoided emissions in cap-and-trade programs and determining
FIGURE 7.2: Components of C&S savings estimate
Naturally
Market Occurring Jr
Baseline Market Adoption Allocation
(NOMAD)
Energy Pot ntial Gross Net Net C&S 7
Use Standards Standards Standards Program Savings
Baseline Energy Energy Energy Savings by
L_ Savings Savings Savings L
5Sa
Unit
nergy Compliance
vings Rate
Source:California Public Utilities Commission(CPUC).(April 9,2010).Codes&Standards(C&S)Programs Impact Evaluation:California Investor Owned Utilities'Codes and Standards
Program Evaluation for Program Years 2006-2008.CALMAC Study ID:CP00030.06 Final Evaluation Report.Prepared by KEMA,Inc.;The Cadmus Group,Inc.;Itron,Inc.;Nexus Market
Research,Inc.;and ENRG,Inc.for the CPUC Energy Division.www.calmac.org/publications/Codes—Standards_Vol It/ FinalEvaluationReportUpdated 04122010.pdf.
7-28 www.seeaction.energy.gov December 2012

whether the avoided emissions data are just for informational associated with steam systems,water pumping,energy recovery,
purposes or are part of a benefit-cost analysis or regulatory require- boilers,chillers,street lighting,district heating,metals production,
ment.While avoided emissions can be part of the benefits of an and agriculture)and households(retrofits and new construction
energy efficiency program,end-use energy efficiency activities can projects associated with cook stoves,water pumping,water purifica-
also be part of a GHG mitigation program. tion,refrigerators,lighting,and whole-building projects).These
methodologies and other tools for documenting GHG reductions
End-use energy efficiency represents a significant,cost-effective from energy efficiency projects are available at the CDM website.
approach to reducing GHG emissions.This has been well documented
in studies by the Intergovernmental Panel on Climate Change and Unlike the CDM,which is regulated and part of a mandatory GHG
numerous other government,private,and non-governmental organi- emissions program,the VCS is a greenhouse gas accounting program
zations.92 However,as with the implementation of energy efficiency used by projects around the world to verify and issue carbon credits
programs solely for the purpose of energy use and demand reduc- in voluntary markets. It is one of several voluntary GHG account-
tions,energy efficiency for GHG mitigation suffers from the difficulty ing programs in place throughout the world.Another is the Gold
of cost-effectively documenting the benefits of energy efficiency Standard,which is a carbon-offset standard that certifies compliance
activities at levels of certainty that can satisfy regulators,investors, credits created through the CDM and voluntary carbon credit pro-
and project proponents. grams.The VCS and the Gold Standard also have methodologies that
can be used to document emission reductions from energy efficiency
Greenhouse gas voluntary and mandatory mitigation programs in activities.These methodologies,as well as the CDM methodologies,
the United States and internationally include energy efficiency as a can be useful tools for evaluators of energy efficiency programs seek-
mitigation strategy.Examples include the Verified Carbon Standard ing to document GHG and other emission reductions.
(VCS,http://v-c-s.org),the Gold Standard(www.cdmgoldstandard.
org),the Regional Greenhouse Gas Initiative(RGGI,www.rggi.org), Investing GHG Funds in Energy Efficiency
and the Clean Development Mechanism(CDM, http://cdm.unfccc. There are at least two ways to think about the relationship between
int/index.html). In these programs,end-use efficiency is either an end-use efficiency and GHG reductions:(1)assume that energy
eligible activity for generating emissions offsets(as in VCS,Gold efficiency is an important societal goal and ask whether GHG
Standard,and CDM)or a strategy for investing funds generated reduction programs can lend value to its attainment;or(2)focus
through the program(as with the use of allowance auction funds on GHG reduction as the goal and ask whether accelerated energy
in RGGI). efficiency is a central element in attaining those essential reductions.
While both goals—efficient use of energy and climate mitigation—
GHG Offsets are important,in the second context,even though energy efficiency
The CDM is the world's largest GHG offset program. It is a is essential to GHG attainment,it does not follow that simply
mechanism under the Kyoto Protocol,which set targets for indus- monetizing GHG emissions will actually call forth the full level of
trialized countries to reduce their domestic emissions as part of energy efficiency that is both attainable and cost-effective,given the
the United Nations Framework Convention on Climate Change.The variety of market barriers associated with energy efficiency.94
basic concept of the CDM is that emissions offsets from projects in
developing countries without a commitment to reduce emissions However,auctioning cap-and-trade allowances can raise,in aggregate,
are sold to entities in developed countries with emissions-reduction large amounts of funds that can be used in complementary programs
commitments at a cost less than these entities'costs to reduce their that do address these barriers.95 These funds can be used in a wide
own emissions.The money generated from the sale of offsets pays for range of efficiency support activities such as public benefit programs;
projects in the developing countries.Greenhouse gas emissions are energy efficiency resource standards;appliance efficiency standards;
reduced in an economically efficient manner,and developing coun- building energy codes;and research,development,and demonstration.
tries receive funds to invest in sustainable development projects.93
One example of this approach is what was done the Regional
As part of the CDM implementation,methodologies(per the Greenhouse Gas Initiative(RGGI),which was the first market-based
terminology of this guide,project-specific M&V plans)have been regulatory program in the United States to reduce greenhouse gas
created for documenting GHG reductions from a wide range of emissions.It is a cooperative effort among the states of Connecticut,
energy efficiency activities.These cover commercial,industrial, Delaware,Maine,Maryland,Massachusetts, New Hampshire,New
and municipal facilities(retrofits and new construction projects York, Rhode Island,and Vermont.Together,these states have capped
December 2012 www.seeaction.energy.gov 7-29

REGIONAL GREENHOUSE GAS INITIATIVE
In 2009,ten Northeastern and Mid-Atlantic states began the strategic energy"purposes,some states contributed a much
Regional Greenhouse Gas Initiative(RGGI),the country's first larger amount to those ends. But from a strictly economic
market-based program to reduce emissions of carbon dioxide perspective,some uses of proceeds clearly deliver economic
(CO2)from power plants.There are now nine states in the returns more readily and substantially than others. For example,
program. RGGI-funded expenditures on energy efficiency depress regional
electrical demand, power prices,and consumer payments
RGGI Produced New Jobs for electricity.This benefits all consumers through downward
Taking into account consumer gains,lower producer revenues, pressure on wholesale prices,even as it particularly benefits
and net positive macroeconomic impacts,RGGI led to overall job those consumers that actually take advantage of such programs,
increases amounting to thousands of new jobs over time.RGGI implement energy efficiency measures,and lower both their
job impacts may in some cases be permanent;others may be overall energy use and monthly energy bills.These savings stay
part-time or temporary.But according to our analysis,the net in the pockets of electricity users directly. But there are also
effect is that the first three years of RGGI led to over 16,000 new positive macroeconomic impacts as well:the lower energy costs
"job years,"with each of the original ten states showing net job flow through the economy as collateral reductions in natural gas
additions.Jobs related to RGGI activities are located around the and oil in buildings and increased consumer disposable income
economy,with examples including engineers who perform effi- (from fewer dollars spent on energy bills),lower payments to
ciency audits;workers who install energy efficiency measures in out-of-state energy suppliers,and increased local spending or
commercial buildings;staff performing teacher training on energy savings.Consequently,there are multiple ways that investments
issues;or workers in state-funded programs that might have been in energy efficiency lead to positive economic impacts;this
cut had a state not used RGGI funds to close budget gaps. reinvestment thus stands out as the most economically benefi-
cial use of RGGI dollars.
Energy Efficiency in RGGI
The states have used their RGGI dollars very differently, in ways Source:Hibbard,P.;Tierney,S.;Okie,A.;Darling,P.(November2011).
that affect the net benefits within the electric sector and in the The Economic Impacts of the Regional Greenhouse Gas Initiative on Ten Northeast
and Mid-Atlantic States,Review of the Use of RGGI Auction Proceeds from the
larger state economy.While all states originally committed to First Three-YearCompliance Period.The Analysis Group.www.analysisgroup.com/
using at least 25%of auction proceeds for"public benefit or uploodedfiles/publishing/articles/economic_impact_rggi_report.pdf.
and will reduce COz emissions from the power sector 10%by 2018.
States sell nearly all emissions allowances through auctions and
invest proceeds in consumer benefits such as energy efficiency,
renewable energy,and other clean energy technologies.
The sidebar"Regional Greenhouse Gas Initiative"summarizes
the results of an impact evaluation of the benefits of the RGGI
investments in energy efficiency.As of the writing of this guide,the
State of California is also reviewing whether to use its new cap-and-
trade GHG program allowance proceeds in a similar manner to the
RGGI.These programs and their evaluations represent new opportu-
nities for energy efficiency and energy efficiency evaluations.
7-30 www.seeaction.energy.gov December 2012

Chapter 7: Notes
68 Electricity system operators are Independent System Operators 75 Note the counterintuitive implication of this standard definition.
(ISO)or Regional Transmission Organizations(RTO)that operate Low precision values correspond to narrow intervals and,hence,
a regional energy market,capacity market,or both.A summary describe tight estimates.This can lead to confusion when estimates
document on the subject of energy efficiency and forward capacity are described as having"low precision."
markets is Gottstein, M.;Schwartz,L.(2010). The Role of Forward
Capacity Markets in Increasing Demand-Side and Other Low- 76 Although there is a close relationship between confidence and
Carbon Resources Experience and Prospects. Regulatory Assistance precision,they are not direct complements of each other. If the
Project(RAP).www.raponline.org/docs/RAP_Gottstein_Schwartz_ confidence level is 90%,there is no reason that the precision needs
RoleofFCM_ExperienceandProspects2_2010_05_04.pdf. to be 10%.It is just as logical to talk about 90/20 confidence and
precision as 90/10.
69 DEER contains information on selected energy-efficient technolo-
gies and measures.It provides estimates of the energy-savings 77 National Action Plan for Energy Efficiency.(2007).Guide to
potential for these technologies in residential and nonresidential Resource Planning with Energy Efficiency. Prepared by Snuller Price
applications.The database contains information on typical measures et al., Energy and Environmental Economics, Inc.(E3).www.epa.gov/
and data on the costs and benefits of more energy-efficient mea- cleanenergy/documents/suca/resource_planning.pdf.A companion
sures.www.deeresources.com. document to the original version of this guide,which addresses
resource planning and energy efficiency.
70 Vine, E.; Hall,N.; Keating, K.M.; Kushler, M.;Prahl,R.(August
2011)."Emerging Evaluation Issues Revisited! 2011 International 78 American Council for an Energy Efficient Economy(ACEEE).
Energy Program Evaluation Conference Proceedings,August 16-18, (October 2011).State Energy Efficiency Resource Standard(EERS)
2011, Boston, Massachusetts. Madison,Wisconsin: International Activity. http://aceee.org/files/pdf/policy-brief/State EERS Summary
Energy Program Evaluation Conference(IEPEC).www.iepec.org/ October 2011.pdf.EERS and other energy efficiency policies are
20llPapersTOC/papers/096.pdf#page-1. described in Section 2.2.1.An up-to-date list of states that cur-
rently maintain an EERS can also be found in the Database of State
71 Skumatz,L.A.;Khawaya, M.;Colby,J.(2009).Lessons Learned Incentives for Renewable Energy(DSIRE)at www.dsireusa.org.
and Next Steps in Energy Efficiency Measurement and Attribution:
Energy Savings,Net to Gross,Non-Energy Benefits,and Persistence 79 Barbose,G.;Billingsley;M.;Goldman,C.; Hoffman,I.;Schlegel,
of Energy Efficiency Behavior. Prepared for the California Institute J.(August 2012)."On a Rising Tide:The Future of U.S. Utility
for Energy and Environment(CIEE)Behavior and Energy Program. Customer-Funded Energy Efficiency Programs."2012 ACEEE Summer
http://uc-ciee.org/downloads/EEM_A.pdf. Study Proceedings;August 12-17,2012, Pacific Grove,California.
Washington, D.C.:American Council for an Energy-Efficient Economy
72 One such study is Hirst, E.;White, D. (1985).Indoor Temperature (ACEEE). LBNL-5755E.www.aceee.org/files/proceedings/2012/data/
Changes After Retrofit:Inferences Based on Electricity Billing Data papers/0193-000173.pdf.
for Nonparticipants and Participants in the BRA Residential Weath-
erization Program. Oak Ridge National Laboratory.www.ornl.gov/ 80 Categories based on:Skumatz,L.A.(1997)."Recognizing All
info/reports/1985/3445600444954.pdf. Program Benefits: Estimating the Non-Energy Benefits of PG&E's
Venture Partners Pilot Program(VPP)."Proceedings of the 1997
73 California Public Utilities Commission(CPUC).(April 2006). Energy Evaluation Conference.August 27-29,1997,Chicago,Illinois.
California Energy Efficiency Evaluation Protocols:Technical, Madison,Wisconsin:International Energy Program Evaluation
Methodological,and Reporting Requirements for Evaluation Conference(IEPEC).www.iepec.org/1997PapersTOC/papers/033.pdf.
Professionals.www.calmac.org/events/EvaluatorsProtocols_Final_
AdoptedviaRuling_06-19-2006.pdf. 81 Skumatz,L.A.;Khawaja, M.S.; Krop,R.(May 2010).Non-Energy
Benefits:Status, Findings,Next Steps,and Implications for Low
74"Uniform Methods Project."(2012). U.S.Department of Energy. Income Program Analyses in California. Revised Report. Prepared for
wwwl.eere.energy.gov/deployment/ump.html. Includes information Brenda Gettig,Sempra Utilities.www.liob.org/docs/LIEE Non-Energy
on steps to increase the accuracy of evaluation results,particularly Benefits Revised Report.pdf.
with respect to sampling.
December 2012 www.seeaction.energy.gov 7-31

82 Water savings can come from energy efficiency measures,such as 91 Some DR programs might actually be set up with participants
low-flow showerheads and cooling tower upgrades that are designed receiving a discounted cost of energy throughout the year,and then
to save energy but also reduce water(and wastewater flows). In penalties if they do not reduce their demand when called upon
addition,there are water-saving projects(reducing irrigation needs, to do so.
process water requirements)that save energy via reduced water
pumping,conditioning,and/or heating. 92 Intergovernmental Panel on Climate Change(IPCC).(2007).
Pathways to a Low-Carbon Economy:Version 2 of the Global
83 In fact,in a strict sense,when used in association with incremental Greenhouse Gas Abatement Cost Curve. Prepared by McKinsey
savings for programs or measures,NEBs should be described as"net" &Company.www.epa.gov/statelocalclimate/documents/pdf/
of what would have occurred absent the energy efficiency programs; mckinsey_summary_11-19-09.pdf.
thus,in some cases they might be positive in an absolute sense,but Intergovernmental Panel on Climate Change(IPCC). (2007). Contri-
negative compared to,for example,what would have occurred if the bution of Working Group III to the Fourth Assessment Report of the
consumer had purchased a standard energy efficiency product. Intergovernmental Panel on Climate Change,2007.www.ipcc.ch/
84 Skumatz,L.A.;Khawaja,M.S.; Krop,R.(May 2010).Non-Energy publications_and_data/ar4/wg3/en/contents.html
Benefits:Status,Findings,Next Steps,and Implications for Low 93 Schiller,S.R.(August 2011)."Documenting Greenhouse Gas
Income Program Analyses in California. Revised Report. Prepared for Emission Reductions from End-Use Efficiency Activities in the Kyoto
Brenda Gettig,Sempra Utilities.www.liob.org/docs/LIEE Non-Energy Protocol's Clean Development Mechanism Offset Program."2011
Benefits Revised Report.pdf. International Energy Program Evaluation Conference Proceedings,
ss As an example of a quantifiable definition of"comfort,"ANSI/
August 16-18,2011, Boston, Massachusetts.Madison,Wisconsin:
ASHRAE Standard 55-2010 defines a range of indoor thermal International Energy Program Evaluation Conference(IEPEC).
environmental conditions acceptable to a majority of occupants. www•iepec.org/20llPapersTOC/papers/115.pdf#page=l.
86 Or when the"perception"is the factor that affects the benefit 94 Schiller,S.R.; Prindle,B.;Cowart,R.; Rosenfeld,A.H.(2008).
(such as comfort),or the associated perception of a return on "Energy Efficiency and Climate Change Mitigation Policy."2008
investment that may affect program"uptake." ACEEE Summer Study Proceedings.August 17-22,2008, Pacific
Grove,California.Washington, DC:American Council for an Energy-
87 Kushler, M.; Nowak,S.;Witte, R(February 2012).A National Efficient Economy(ACEEE).www.aceee.org/files/proceedings/2008/
Survey of State Policies and Practices for the Evaluation of Ratepayer- data/papers/8_306.pdf.
Funded Energy Efficiency Programs.American Council for an Energy-
Efficient Economy(ACEEE). Report Number U122.www.aceee.org/ 95 Under acap-and-trade program,an overall emission tonnage cap
is set for an affected sector or set of emission sources.Governments
resea rch-report/u 122.
create allowances representing the temporary right to emit one unit
88 State and Local Energy Efficiency Action Network.(May (e.g.,one ton)within the total cap amount.Initial allowance alloca-
2012).Evaluation, Measurement and Verification(EM&V)of tions can be sold(auctioned)by the government or distributed for
Residential Behavior-Based Energy Efficiency Programs:Issues and free to affected sources or to others.In the above-referenced paper
Recommendations. Prepared by Todd,A.;Stuart,E.;Schiller,S.; (Schiller et.al.),the authors call the activities paid for by the allowance
Goldman,C.; Lawrence Berkeley National Laboratory.wwwl.eere. money complementary policies and programs—in other words,pro-
energy.gov/seeaction/pdfs/emv_behaviorbased_eeprograms.pdf. grams that while running parallel to cap-and-trade programs are still
integral to the success of cap-and-trade and GHG mitigation policies.
89 A panel data model is an analysis model in which many data points
over time are observed for a certain population(also called a time-
series of cross-sections).
90"Building Energy Codes Program."(2012).U.S. Department of
Energy.www.energycodes.gov.
7-32 www.seeaction.energy.gov December 2012

Chapter 8
Impact Evaluation Planning
Chapter 8 builds on preceding chapters and presents impact evaluation concepts and the steps involved in the planning
process.These include the development of evaluation approaches, budgets, and schedules.The first section discusses
how evaluation planning and reporting is integrated into the program implementation process, while the second section
presents the concept of a hierarchy of planning documents, starting with an evaluation framework.The third section
presents 14 issues and questions that help determine the scope and scale of an impact evaluation.The last section
provides some guidance on preparing program impact evaluation plans and site-specific measurement and verification
(M&V) plans, as well as a checklist for preparing an evaluation plan.
8.1 INTEGRATION OF EVALUATION INTO THE Figure 8.1 shows the energy efficiency program implementation
PORTFOLIO IMPLEMENTATION CYCLE cycle,emphasizing evaluation activities and feedback to the cur-
Before describing the evaluation planning process,it is important rent and future programs(portfolio).This figure and the following
to understand how it is integral to what is typically a cyclic program discussion concerning evaluation timing are related to the activities
or portfolio planning-implementation-evaluation process. In most associated with formal impact evaluations,usually conducted by
cases,the overall cycle time frame is determined by a program s—or
an independent third-party evaluator.There are other documenta-
more likely a portfolio's—funding and contracting schedules(cycles). tion activities that are carried out by program implementers.These
These portfolio implementation cycles can be one or two years or include defining baselines,conducting project inspections,and calcu-
even longer.The point at which programs are being designed is lating savings for the paying of incentives to contractors and prepar-
ideally when the impact,market,and process evaluation planning ing their own(implementer)estimates of claimed savings.There is
process should begin.This is primarily so that the program budget, an interaction between these different activities by the implementer
schedule,and resources can properly take into account evaluation and evaluator,which is discussed in Section 8.3; however,this text is
requirements and opportunities. It is also a way to reinforce the primarily focused on the activities carried out by just the evaluator.
concept that evaluation is an integral part of the portfolio process, The first three evaluation activity steps displayed in Figure 8.1 are
supporting the portfolio's success through an assessment of the pro- further described below:
gram's impacts as well as the program's theory for how savings are Program goal setting.When a program is first envisioned,often
to be achieved.961n addition,early consideration of evaluation,on a as part of a portfolio of programs,is when both program goals
very practical level,helps ensure that the data collection required to and evaluation goals should be considered. For example,if the
support expected evaluation efforts is accommodated at the time of program goal is to save electricity during peak-use periods,
implementation. It is also helpful to decide early in the process which the evaluation goals can include accurately documenting how
entities will collect and analyze data(e.g.,program implementers,an much electricity is saved during a defined peak period(gross
independent third-party evaluator,or both). impact)and how much of these savings can be attributed to
the program(net impact). High-level evaluation schedules and
Evaluations should be produced within a portfolio cycle or very budgets can also be established at this time.
soon after the completion of a cycle.This is so evaluation results Program design.The evaluation design effort should begin
can document the operations and effects of the programs in a timely during the program design phase.The objective should be a
manner and provide feedback for ongoing program improvement, preliminary evaluation plan that includes a schedule and a
provide information to support energy efficiency portfolio assess- budget as well as a description of each work task to be com-
ments(including market assessments and potential studies),and pleted.The evaluation issues described in Section 8.3 should
help support the planning of future portfolio cycles,load forecasts, be raised,although not necessarily fully addressed,at this time.
and energy resource plans. For impact evaluations that examine Whereas a program design is usually completed at this stage,it
energy savings of certain measures and program mechanisms,the is likely that the evaluation plan will not be fully defined.This is
evaluation information can also be used to inform deemed savings typically because of the necessary priority given to the program
values though updating of technical reference manuals(TRMs). design and the need to have the program plan pretty well
December 2012 www.seeaction.energy.gov 8-1

completed before the evaluation plan can be finalized. the evaluation planning process should be started well before
Another reason why the evaluation design may very well not a program is launched.Trying to collect baseline data on equip-
be completed at this time is that it is not unusual,although ment that has been replaced(and usually thrown away)after
not best practice,to select the evaluator after the program has a project is completed can usually be ranked between difficult
been fully designed(and sometimes after it is implemented). and impossible.
Nonetheless,it is ideal to assign evaluation staff or consultants
to the program evaluation during program design activities. The overall evaluation plan should be reviewed with program
Regardless of when the formal evaluator gets involved,evalua- implementers and,equally important,with the appropriate oversight
tion goals and metrics should be considered,and an initial set of body(ies),to ensure that it meets the requirements of policymakers,
evaluation priorities(with a rough schedule and budget)should portfolio managers,and/or regulators.These requirements are hope-
be established during this portfolio cycle phase. fully set in the evaluation framework,which is a high-level document
• Program launch.Program launch is when program materials prepared outside of the portfolio cycle as a guiding document for the
and timing strategies are finalized and made ready,implemen- relevant jurisdictions.This framework document concept is discussed
tation contracts(if needed)are negotiated,trade allies and key in Section 8.2.
stakeholders are notified,and materials and internal processes Although it often is preferable to start an evaluation prior to the
are developed to prepare for program introduction and program launch in order to collect baseline information,in most
implementation. If the detailed program evaluation plan has cases,either(1)the evaluation and program start simultaneously
not been prepared,it should be prepared before the program due to the typical interest in initiating a program as soon as possible,
is launched—or if not,soon after it is launched.An outline of or(2)the evaluation starts well after the start of the program due to
such a plan is presented in Section 8.4. serial planning of programs first and then evaluation in the portfolio
• It is in the evaluation plan that the program-specific evaluation implementation cycle.In general,the sooner the evaluation process
issues are fully addressed and resolved, including specifying can begin,the higher the quality of the evaluation and the sooner
the data needed to perform the evaluation as well as which information—such as indications of where programs are meeting,or
data should be collected during program implementation. not meeting,their goals—can be provided.
This is also the time when some baseline data collection can
take place, but most likely only after the program's projects In terms of reporting,evaluation information can be summarized
and participants are identified. In situations where project site and provided on any time cycle,but most cycles are driven by(1)
baseline data collection is required as part of the evaluation, the time required to complete the evaluation work and prepare
FIGURE 8.1: Program implementation cycle with high-level evaluation activities
schedule,ACTIVITY: ACTIVITY: ACTIVITY: ACTIVITY-
Program Goal Setting Program Design Program Launch Implement Evaluation
Evaluation Activity: : =,A
Evaluation Activity Evaluation Activity: Evaluation Activw
,ft2ts. Prepare preliminary Prepare detailed Implement evaluation
evaluation plan evaluation . .
collectand reporting and baselineexpectations data as --.-.
FOR •
PROGRAMS •
8-2 www.seeaction.energy.gov December 2012

reports after the last projects in a program cycle are completed,(2) EVALUABILITY
regulatory needs,and(3)a desire to get the information needed
to implementers so they can adjust existing programs and design "Evaluability"is an assessment of the probability that evalua-
new ones(that may start right after the completion of the current tion information will be available when evaluations are
program cycle)using current and relevant evaluation,measurement, actually undertaken.Some data(e.g.,the age of a building)
and verification(EM&V)information. For future program designs, can be gathered at any time;some data(e.g.participant spill-
projected savings estimates can be adjusted based on program over,current hours of operation)are best gathered at the time
evaluation results.Assumptions underlying the energy efficiency of evaluation;and some data must be gathered at the time of
potential analysis used for planning at the beginning of the program implementation or they will be lost forever or rendered unreli-
cycle can then be updated based on the full net impact analysis. able due to changes in personnel or fading recollection(these
These data then feed back into the goal setting and savings potential include free ridership,removed equipment,or non-participant
analysis activities,and the cycle repeats to allow for an integrated customer contact).The list below is an example of some of the
planning process for future programs. items included in an evaluability assessment template:
8.2 HIERARCHY OF EVALUATION PLANNING • Is there a way to track participants?
AND REPORTING DOCUMENTS • Is there a way to track non-participants?
• Are specific locations of measures being tracked?Can
There are several evaluation planning and reporting documents that they be found?
are typically prepared as part of the evaluation effort.The following is • Are program assumptions being tracked on a site-specific
an overview of these documents in the context of a portfolio of pro- level (e.g.,hours of operation)?
grams that uses public or utility customer funds and has a program . Is the delivered energy-saving service and/or installed
administrator with some government(regulatory)oversight.While retrofit being recorded?
privately funded portfolios can use many of the concepts presented Does the device recording savings include the outcome
here,the formality and requirements for independent evaluation may or result of the activities?
be different.Publicly funded portfolios usually have more stringent . Are savings assumptions documented?
reporting requirements than privately funded ones.
• Is the source of savings assumptions specified?
8.2.1 Evaluation Planning Documents • Are the pre-retrofit or baseline parameters being recorded?
Entities such as states,regulatory commissions,utilities,and others • Does the database record the"as-found"values for
can establish and document their evaluation requirements in a parameters used to estimate ex ante(projected)savings?
hierarchy of documents. Figure 8.2 outlines the hierarchy of these • Does baseline monitoring need to take place?
documents and indicates their typical time frame(or time horizon) • Can one of the impact evaluation methods specified in
and applicability level(e.g.,state or utility program administra- this guide be used?
tor,program or portfolio of programs,or individual projects).The • Are there code compliance or program overlap issues
increasing level of detail in this hierarchy of documents provides for savings estimation?
indications of the appropriate place for each stakeholder in the
overall energy efficiency effort to provide input.For example,all
stakeholders may be interested in an EM&V framework but—on content and scope of other EM&V documents(e.g.,annual
the other end of the spectrum—only program implementers and portfolio and statewide evaluation reports).The issues that
evaluators will generally be concerned with the details of a program would be considered when preparing such a framework are
research or M&V plan. described in Section 8.3.This is perhaps the principle document
that all stakeholders can focus on and use to provide high-level
Descriptions of the documents are as follows: input—the"forest versus the trees"of an EM&V infrastructure.
• EM&V Framework.A framework is a primary document that Portfolio Cycle EM&V Plan.This is the planning document that
lays out EM&V principles,metrics,allowable approaches, indicates which major evaluation activities will be conducted
net versus gross savings issues, reporting requirements and during the evaluation cycle(typically one,two,or three years)
schedules,and the roles and responsibilities of various entities. and describes them at a high level,including budget and alloca-
An EM&V framework document tends to be"fixed" but can tion of resources and effort between programs,measures,
be updated periodically. It often sets the expectations for the market sectors,etc. It lists all the planned, major evaluation
December 2012 www.seeaction.energy.gov 8-3

FIGURE 8.2: Hierarchy of EMN planning documents Protocol (IPMVP)measurement and verification options
are applied,orjust for inspections if deemed savings values
TIMEFRAME COVERAGE are to be used.
Multiple EM&V Region, State,
Year bm FRAMEWORK or Program Jurisdictions,such as state or regional organizations oversee-
Administrator ing energy efficiency portfolios,should also seriously consider
developing TRMs.The contents of these TRMs vary,but they
PORTFOLIO Region, State, are typically a database of standardized,state-or region-specific
AnnualCYCLE EM&V or Pro
Multiple Year gram algorithms(deemed calculations)and associated savings estimates
PLAN Administrator (deemed savings values)for conventional electric and natural gas
energy efficiency measures.The TRMs are used by energy efficiency
As Required . . Program program administrators and implementers to reduce EM&V costs
(e.g., annual) or Portfolio and uncertainty in how measured savings will be credited with value
as they prepare their projected and claimed savings values.Similarly,
FM evaluators will also use the information in TRMs to prepare their
As Required M&V Project evaluated savings values.TRIM values for individual measures are
(e.g., annual) PLAN or Site not always formally vetted in a regulatory process,although this is a
good practice.See Section 4.2 for more information on TRMs.
Source:Schiller,S.R.;Goldman,C.A.(August2011).'Developing State and National In some cases,different program administrators(e.g.,utilities)in the
Evaluation Infrastructures—Guidance for the Challenges and Opportunities of EM&V."
2011International Energy Program Evaluation Conference Proceedings,August16-18, same state or region will use different savings values for the same
2011,Boston,Massachusetts.Madison,Wisconsin:International Energy Program measures.Although different values may be perfectly appropriate
Evaluation Conference(IEPEQ.Www.iepec.org/2o11PopersTOC/papers/o73.pdfapage=l. for the same measures because of different baselines,delivery
mechanisms,weather,or other factors,it is important for each
activities with a schedule for when they are to be conducted. jurisdiction to at least have established and consistent procedures
Examples of such major evaluation activities are the impact, for building and maintaining TRMs,with deemed calculations and
process,and market evaluations,as well as updates to any savings values that can be accepted for projected,claimed,and
technical reference manuals(TRMs).This EM&V plan might evaluated savings estimates.State regulators and program adminis-
indicate,for example, (1)which programs would have rigorous trators may also want to consider pooling their resources to support
evaluations in each year and which would only have verification development of regional TRMs as well as statewide or regionally
reviews,and(2)which programs will undergo process evalua- coordinated TRM updating processes and schedules.Examples
tions and which will not. of this are the Regional Technical Forum in the Northwest(www.
• Evaluation Activity-Specific Detailed Research Plans. nwcouncil.org/energy/rtf)and the Regional EM&V Forum in the
Research plans are created for the major EM&V activities or northeast and mid-Atlantic regions(http://neep.org/emv-forum).
studies planned in a given cycle prior to the time each effort Other TRMs available online are listed in Appendix C.
is launched. Examples of these plans are(1) program-specific 8.2.2 Evaluation Reports
impact evaluation plans that go into substantial detail on what As described in Chapter 3,there are impacts that are reported by
evaluation approaches will be used,schedule, budgets,the the program administrators or implementers,and there are impacts
data that will be collected,and results that will be reported; reported by the evaluator.Savings estimates for programs tend to be
and (2)process,market effects,and market baseline study reported in one or more of the following classifications:
plans that would similarly provide sufficient detail to guide the
actual implementation of the evaluation activity. • Projected savings:values reported by a program implementer
or administrator prior to the time the energy efficiency
• Project-Specific M&V Plans. Project-specific plans may be activities are completed
required for projects that are part of a custom program and
that are selected for analysis and inspection.These are the • Claimed savings:values reported by a program implementer
plans that describe the specific activities that would be or administrator after the energy efficiency activities have
conducted at a single site when one or more of the four been completed
International Performance Measurement and Verification
8-4 www.seeaction.energy.gov December 2012

• Evaluated saving:values reported by an independent third- portfolio cycle evaluation reports to be completed well after the
party evaluator after the energy efficiency activities and impact end of the completion of the evaluated cycle and into the time of
evaluation have been completed. the next cycle.
Evaluators report the evaluation results and,as appropriate,work The above list indicates the typical reports associated with impact
with(1)regulators to assess whether goals have been met,(2) evaluations. However,other reports often prepared by evaluators
program administrators to implement recommendations for current include process evaluations and market assessments.These are
or future program improvements,and/or(3)resource planners to prepared as required to support the implementation of effective
understand the historical role and future role of energy efficiency energy efficiency programs.These reports are also made publicly
as an energy resource. Reporting also provides information to the available,preferably with summaries also provided in a manner that
energy consumers and general public,who are often funding these is accessible to laypersons.
efforts,on what has been achieved with their investments.
8.3 PRINCIPLES AND ISSUES THAT
Correlated with the evaluation planning documents described above, DETERMINE THE SCOPE OF AN IMPACT
the following are the typical types of impact evaluation reports that EVALUATION
are prepared. This section presents and discusses issues and principles that should
• Project-Specific M&V Reports.These reports document the be defined by each jurisdiction in order to develop a framework
impacts determined for a specific project,site,or measure and document for their evaluation activities.As with the last section,it
the methods used to determine the impacts.They tend be provides an overview of these issues and principles in the context
reviewed with the project administrators and implementers of a portfolio of programs that uses public or utility customer funds
before they are finalized,and their results are made available and has a program administrator with some government(regula-
on a limited basis to protect the confidentiality of the con- tory)oversight.And,again,while privately funded portfolios can
sumer information that is usually included. use many of the concepts presented here,the formality and typical
• Impact Evaluation Reports.The results of conducting the requirements for independent evaluation may not be relevant.
evaluation activities described in each impact evaluation plan
are documented in an impact evaluation report.The report Because of differences in approaches to EM&V and no"cross-border
documents the impacts,and perhaps cost-effectiveness,of trading"of energy efficiency savings,it has not been necessary for
a program,as well as the methods used to determine the regulatory commissions and administrators of ratepayer-funded
impacts. Program administrators and implementers usually energy efficiency programs in states to define a saved unit of energy
have opportunities to provide input on these reports.The final in exactly the same way.97 Thus,each jurisdiction(e.g.,state)can,
reports are also usually publicly available. and many do,develop EM&V requirements(and TRMs)that are
appropriate for their own situations,with the EM&V requirements
• Portfolio Cycle Evaluation Reports.The results of carrying out for each jurisdiction linked to the needs of that jurisdiction.
the evaluation activities described in an EM&V portfolio plan
are documented in a portfolio cycle(e.g.,annual or biennial) How the evaluation issues and principles are defined for each juris-
evaluation report.It documents the impact metrics(e.g.,gross diction depends on the specific programmatic and regulatory context
and net energy and demand savings,first year,and lifetime)and (including any mandates)found within each jurisdiction,the objec-
cost effectiveness associated with the portfolio of programs as tives and scale of the energy efficiency activities being evaluated,and
well as the methods used to determine the impacts.Program how EM&V results will be used.For example,one state may have
administrators and implementers also usually have opportunities very limited goals for energy efficiency and may not have perfor-
to provide input on these reports.The final reports are made mance incentives for their energy efficiency portfolio administrator.
publicly available with summaries or synthesis reports provided It also may have a limited level of naturally occurring or mandated
in a manner that is accessible to laypersons and with guidance (via codes and standards)energy efficiency activity.Another state
on context and interpretation of the evaluation findings.Interim may have established aggressive,long-term energy-savings targets
reports are also suggested so that progress indicators can be in legislation,developed a performance-based incentives scheme for
provided and problems,if they exist,are identified before the program administrators,and have high energy costs as well as a need
end of the program cycle. Interim results can also support plan- for very solid savings data for resource planning purposes.The high
ning efforts for the next portfolio cycle,as it is common for the energy cost scenario may also have resulted in a high level of natural
December 2012 www.seeaction.energy.gov 8-5

and mandated energy efficiency activity.Given these differences, 7. What is the reporting"boundary"?Are transmission and
the first state may only need a limited level of EM&V and use fairly distribution (T&D)losses included,and how"granular"will
permissive baselines,while the second state might require very rigor- the results be?See the discussion below in Section 8.3.3.
ous(and expensive) EM&V with well-defined baselines.Although 8. What are the schedules for implementing the evaluation
in both scenarios EM&V can be valuable to regulators and program and reporting?See the discussion below in Section 8.3.4.
administrators and implementers,these kinds of considerations drive
9. What impact evaluation approaches will be used?This simply
the type,style,budget,and timing of EM&V requirements.
relates to selecting one or more of the impact evaluation
These EM&V requirements can be developed by program administra- approaches defined in Chapters 4,5,and 6 and Appendix B
tors,agencies with responsibility for overseeing EM&V activities(e.g., (top-down evaluation)that will be allowed,or preferred,for the
state utility commissions,energy offices),and other stakeholders,and evaluation of different programs or the portfolio as a whole.
then documented in the framework document described in Section 10. What are expectations for savings determination certainty
8.2.Preferably,they all draw from established EM&V principles,such (confidence and precision)?Section 7.4 includes a discussion
as those indicated in Section 7.6.To this end,stakeholders in each of uncertainty and managing and defining the risk of savings
jurisdiction should work together to address the following 14 issues uncertainty.
regarding EM&V.These issues below have been vetted in a number of 11. Which cost-effectiveness tests will be used?Cost-effectiveness
jurisdictions and form a good basis for defining the needs of regula- test selection determines what data collection will be required
tors,administrators,implementers,and resource planners. However, as part of the impact evaluations.Cost-effectiveness tests are
the list should be reviewed and customized for each jurisdiction,and briefly described in Appendix B.
to meet exigent regulatory needs. 12. How are evaluated savings estimates applied—looking back/
1. What are the evaluation objectives,metrics,and research going forward?See the discussion below in Section 8.3.5.
issues that support program policies and/or regulations? 13. What are the data management strategies?See the discussion
Driving the response to this question is the need to know the
below in Section 8.3.6.
policy and/or regulatory goals that are the basis for the energy
efficiency programs and thus the evaluation of the programs. 14. How are disputes addressed?See the discussion below in
See Section 8.3.1.and Section 2.1.1 for discussions of energy Section 8.3.7.
efficiency program policy contexts. These issues are presented in what can be considered a linear
2. What are the evaluation principles that drive the effort? sequence,but many are interrelated and the overall planning pro-
See Section 7.6 for a discussion of evaluation characteristics cess is certainly iterative.These issues can be addressed through a
and principles. variety of mechanisms,such as collaborative efforts(as is the case in
3. What is the scale of the evaluation effort?How much time, Washington),with advisory groups(as is the case in New Mexico),or
effort,and money will be spent on evaluation?What is the with regulatory proceedings(as is the case in many states).The end
relative allocation of resources among impact,market,and result of addressing the above 14 issues is the evaluation framework
process evaluations?Section 7.5 includes a discussion on that documents the evaluation expectations for all stakeholders.
defining the scale of the effort and establishing related budgets. Experience has indicated that,if the funding and time requirements
4. Who will conduct the evaluations,how is an independent
for reliable evaluations are understood and balanced with informa-
li
evaluation defined,and what are the relative EM&V roles on needs and accuracy expectations and approved by the primary
stakeholders,evaluation efforts can be well-supported and succeed
between implementers,evaluators,regulators,stakeholders,and others?See the discussion below in Section 8.3.2. in providing the results desired.
5. Is performance determined on the basis of net or gross 8.3.1 Defining Evaluation Objectives and Metrics
savings?What factors are included in defining net savings? Impact evaluations should focus on a program's performance
Chapter 5 includes a discussion of net savings definitions, at meeting its key implementation goals and,if desired,provide
uses of net versus gross savings,and issues associated with information for future program and resource planning.To this end,
determining and using net savings. program managers and regulators need to be assured that the evalu-
6. What are the baselines against which savings are determined? ations conducted will deliver the type and quality of information
Section 7.1 includes a discussion of baselines and their needed.This requires consideration of the specific program benefits
influence on impact evaluations. (metrics)to be evaluated and reported,the concepts behind the
8-6 www.seeaction.energy.gov December 2012

results the program is expecting to achieve(usually referred to as the • Inform decisions regarding program administrator compensa-
program theory),and whether any other related evaluations will be tion and final payments(for regulated programs and
concurrently conducted and coordinated. performance-based programs)
Under-designed evaluations can waste valuable resources by not • Assess if there is a continuing need for the program
reliably providing the information needed—or worse,providing incor- • Provide specific data for demand forecasts and resource
rect information.Related to under-designed evaluations are delays that planning in an integrated resource planning(IRP)effort(in the
can make it impossible to collect valuable baseline data and postpone format and with baselines and metric definitions consistent
the results so that they cannot be used for current program improve- with the resource planners' requirements).
ment or future program design.Evaluations can also be over-designed,
In practice,the selection of evaluation objectives will be shaped
addressing issues that are not priority issues or employing methods by many situational factors. Probably the most important factors
that could be replaced by less-costly approaches.Evaluation activities
are the program goals—hopefully goals that are well defined and
should be prioritized so that evaluation resources—typically limited—
quantifiable into evaluation metrics.Some program goals,beyond
can be focused on the issues of importance.Like most activities,an
numerical metrics of energy and demand savings,cost effectiveness,
evaluation that is well defined in terms of objectives and scale,and also
and maximizing energy or peak savings within portfolio budgets,are
affordable,is more likely to be completed successfully than one with
as follows:
undefined or unrealistic objectives and budget requirements.
• Maximize leverage of portfolio dollars in creating private
Setting impact evaluation objectives involves defining the specific investment in energy-efficient products to achieve savings
information(metrics)that will be reported from the evaluation.The . Defer specific resource planning needs(e.g., peaking or
scale of the evaluation is a more general concept,indicating how baseload power plants,transmission or distribution investments)
much effort(e.g.,time,funding,human resources)will be expended
on the evaluation.It involves balancing the various needs for Reduce greenhouse gas(GHG)emissions
information,accuracy of the information,and timing of the informa- • Maximize the fraction of the population participating in
tion against the available resources,including funding. portfolios by sector
• Maximize consumer satisfaction.
8.3.1.1 Evaluation Objectives
As discussed in the beginning of this guide,evaluations can have In Section 2.1.1,three common examples of program goals are
three primary objectives: briefly described. For those three examples,the following are some
comments on evaluation objectives:
1. To document the benefits/impacts of a program and determine
• Cost effectiveness.All cost-effective energy efficiency evalu-
whether the program(or portfolio of programs)met its goals
ations for programs with this policy context have to focus
2. To help understand why program-induced effects occurred and resources on determining the long-term cost effectiveness
identify ways to improve current and future programs of the programs. Evaluation methodologies can be complex,
3. To support energy demand forecasting and resource planning by as assumptions about the baselines,net-to-gross ratios,
understanding the historical and future resource contributions and non-energy benefit factors used in program evaluations
of energy efficiency compared to other energy resources. can be critical to the calculations selected for determining
Therefore,the first step in planning an evaluation is simply picking
cost-effectiveness.
which of these objectives(if not all)are applicable,prioritizing them, Energy efficiency resource standard(EERS).Evaluations for
and making them more specific to the evaluated program.The follow- programs with an EERS policy objective may be able to focus
ing are some typical and more specific impact evaluation objectives: relatively less effort on long-term cost effectiveness, because
• Measure and document energy and peak savings the EERS mandates are often promulgated on the assumption
(which can only be changed by future legislation)that energy
• Document program milestones,such as homes weatherized efficiency will always be less expensive than other resources
or people trained up to the level of the EERS target percentage. Evaluations of
• Measure and document avoided emissions EERS programs thus focus on the year-by-year tally of progress
• Provide data needed to assess cost-effectiveness toward the EERS targets.
• Provide ongoing feedback and guidance to the program Target spending budget.Evaluations for programs with target
administrator budgets have to focus on the most cost-effective use of the
December 2012 www.seeaction.energy.gov 8-7

budgets,so that they can demonstrate that the administrators • Lifetime savings(savings that occur during the effective
are getting the maximum benefit from each program dollar. useful life of the efficiency measure): MWh saved during the
measure's lifetime, in years
Another policy issue that can affect program evaluation budgets is
• Monetary savings:dollars saved in electricity or natural gas
whether the utility or other program administrators are eligible to
costs and maintenance per year.
receive a financial incentive(bonus)for managing a successful pro-
gram portfolio—or suffer a penalty for not meeting the established The metrics listed above are related to energy savings and avoided
goals.The scale of the penalty or bonus can determine the level of emissions.However,other non-energy metrics may also be a subject
rigor required in the evaluation as well as the level of attention paid of the impact evaluation;examples include direct or indirect job
to the evaluation activities. Evaluators should be cognizant of the gains created over the program implementation period for a training
level of intensity that some stakeholders apply to seemingly obscure program or annual water savings associated with a low-flow faucet
issues of evaluation methodology when they can affect program program that is focused on water heating savings.In addition,a focus
goals and consequences,such as administrator incentives. of many programs is transforming markets(see Section 7.10.3 and
However,beyond just what the program goals are and uses of Appendix B),and thus other metrics can include market penetration
the evaluation results,other factors also affect the selection of of energy efficiency products or pricing of such products.
evaluation objectives: As a reminder,as discussed in Section 7.4,evaluation results,like any
• Whether the program is new,expanding,or contracting estimate,should be reported as expected values with an associated
• The policy and/or regulatory framework in which the level of variability.
evaluation results will be reported
8.3.1.3 Other Evaluation Efforts and Other Programs
• The relative priority placed upon the evaluation's
comprehensiveness and accuracy by the responsible While this guide is focused on impact evaluations,there are
authorities(i.e.,the budget and resources available). other types of evaluations(as described in Appendix B). If other
evaluations,such as process or market effects evaluations,are to
One approach to address these considerations is to use theory-based be conducted,their plans should be integrated with the impact
evaluation(TBE). In short,TBE involves defining the events a program evaluation objectives and plan. If cost-effectiveness analyses are to
is intended to cause,along with the precise causal mechanisms lead- be conducted,it is critical to define which cost-effectiveness test(s)
ing to these events,with the evaluation,then focusing on testing the will be used,and thus what impact evaluation data are needed.
consistency of observed events with the overall program theory(and Furthermore,if more than one program is being evaluated,and
thus the goal).Theory-based evaluation is briefly discussed in Section the programs may have some interaction,then the programs,their
7.10.3,in the market transformation program section. evaluations,and the assignment of benefits to one program versus
another need to be coordinated to avoid—or at least minimize-
8.3.1.2 Evaluation Metrics double-counting of savings.
In terms of reporting impact evaluation results,the key parameters
are the units and time frame of the units.Some examples include 8•3.2 Evaluator Roles and Selection
the following: While evaluation is relevant to many kinds of business and govern-
• Electricity savings:kWh saved per year and per month
ment efficiency portfolios,the importance of defining the role and
independence of the evaluator is primarily an issue for programs that
• Demand savings(example 1): kW saved per month of each are funded with public or energy consumer funds.In such programs,
year of program,averaged over peak weekday hours a government regulatory entity,typically a utilities commission,
• Demand savings(example 2): kW savings coincident with oversees the actions of the program administrator.This subsection
annual utility peak demand, reported for each year of addresses the situation in which a government regulator oversees a
the program program administrator's conduct of efficiency activities and wants
• Avoided emissions(example 1): metric tons of COz and some level of independent assessment of program/portfolio impacts.
SOx avoided during each year of the program It first describes the possible roles of evaluators in program imple-
• Avoided emissions(example 2): metric tons of NOx avoided
mentation and evaluation,and then presents criteria for defining
during ozone season months of each year of the program independent third-party evaluators and criteria for their selection.
8-8 www.seeaction.energy.gov December 2012

Starting with impact evaluation—particularly determining energy However,in general practice,"independent third party"is thought
and demand savings—there are two possible sets of results reported to mean that the evaluator has no financial stake in the evaluation
after a project or program is implemented:claimed savings and results(e.g.,magnitude of savings)and that its organization,its
evaluated savings.Claimed savings,those reported by the program contracts,and its business relationships do not create bias in favor
administrator or implementer,are needed not only for possible of or opposed to the interests of the administrator,implementers,
reporting to a government/regulatory agency that oversees program participants,or other stakeholders such as utility customers
efficiency programs(such as utility program administrator reporting (consumers). However,different states'regulatory bodies have taken
to a utilities commission)but also for the basic implementation of different approaches to(1)defining the requirements for evaluators
the programs. For example,on a project-by-project basis,claimed who are asked to review the claimed savings and prepare evaluated
savings need to be determined in order to pay contractors or savings reports,and(2)who hires that evaluator.
consumers whose payments depend on actual implementation
and/or achieved savings.Claimed savings also need to be reported A 2012 American Council for an Energy-Efficient Economy(ACEEE)
internally to justify expenditures,as would be the case with any study indicated the following:98
organization.Staff or consultant evaluators(e.g.,engineers,analysts, .....there is a great diversity among the states in how they
econometricians),working directly for the program implementers handle the evaluation of ratepayer-funded energy efficiency
and/or administrators,almost always prepare these claimed savings. programs.This begins with the administration of the evaluation
Their role is to directly support the program implementation and function itself,where just over a third of states(37%)feature
prepare required internal and external(e.g.,regulatory)reporting. utility[evaluation]administration,36%feature[evaluation]
For the above-listed administrator/implementer functions,those administration by the utility regulatory commission or a
doing the work may or may not have the formal title of evaluators. combination of the commission and utilities,and over a
Evaluator is a very broad term that can describe people conducting quarter(27%)feature[evaluation]administration by some
evaluation activities who are part of a consulting,program admin-
other government agency or third-party entity. Most states
istration,implementation,government,utility,or other type of (79%)rely on independent consultants/contractors to conduct
organization.If not called evaluators,they may be known as analysts, the actual evaluations,although a substantial minority(21%)
engineers,or M&V engineers,or may simply be part of the adminis-
use utility and/or government agency staff.
trator's or implementer's program implementation team. Irrespective of how the relationships are determined or who hires
whom,the objective is for all parties to the evaluation to believe that
Evaluated savings,on the other hand,are only required if some the reported results are based on valid,unbiased information that is
entity,such as a government regulatory agency,wants an indepen- sufficiently reliable to serve as the basis for informed decisions.
dent third party to either determine themselves or double-check the
claimed savings that are determined by the implementer/administra- CLOSING THE LOOP-INTEGRATION OF
tor.This leads to several resulting questions: IMPLEMENTER AND EVALUATOR
• Who is responsible for which evaluation activities?
• Will there be overlap such that an independent third-party There has been a noticeable paradigm shift in evaluation in
evaluator,in preparing evaluated savings,only simply confirms recent years.The old model brought in the evaluator at the
the work done by the implementer/administrator in preparing tail end of the project to assess delivery,cost-effectiveness,
the claimed savings(verification)?Or,will the independent and achievement of stated goals.A different model brings
third-party evaluator conduct some of its own data collection the evaluator in at the onset of the program,as an integral
and impact evaluation analyses? part of the team. Program goals are linked to specific metrics,
• What is meant by an independent third-party evaluator,and which are linked to specific data collection methods.The
who retains them to prepare the evaluated savings reports? evaluator can provide feedback in real time to provide timely
assessment and recommendations,if needed,for corrective
Starting with the last question first,there is no formal definition of actions.This model needs to be balanced with the possible
independent or third-party evaluator,as well as there are no well conflicting nature of evaluation goals—the implementer's
well-established precedents as to who hires the entity(ies)that goal of understanding and improving the program perfor-
provides the evaluated savings reports.The hiring entity could be the mance and a regulating authority's goal of ensuring that the
regulator or the administrator,or perhaps some other entity. savings reported are"real."
December 2012 www.seeaction.energy.gov 8-9

In terms of the first and second questions listed above—who does bring to the project,availability of the key staff,labor rates and
what and whether there are overlaps—there are many options,as overall budget expectation,and approach to the impact evaluation
alluded to in the ACEEE study referenced above. Having complete work,including schedule and deliverables.
analyses conducted independently by a third party provides perhaps
the greatest level of due diligence and integrity for evaluated Budget expectation and approach to the work(instead of a set work
savings values.However,such analyses do add costs,and it is not scope,deliverables,and budget)are listed in these criteria because
uncommon for there to be overlaps in the determination of savings. it is usually not practical to ask bidders to prepare a complete scope/
Also,because a common objective of evaluation is to improve the budget during the proposal process.This is because either the pro-
performance of a program and help with program improvement,a grams to be evaluated are not fully defined and/or the typical first
totally independent approach does not directly favor a tight working task is to prepare the portfolio cycle(e.g.,annual or biennial)EM&V
relationship between the evaluator and the implementer/admin- plan and the evaluation activity-specific detailed research plans,if
istrator.Thus,the selection of an evaluator can require balancing not the evaluation framework itself.
evaluation independence(so that the evaluation is considered Traditionally,evaluation consulting firms tend to use econometricians
objective)with the desire to have the evaluator close enough to the (professionals who apply statistical and mathematical techniques to
process that the evaluation provides ongoing and early feedback problem solving)and engineers.In the last 10 years or so,there has
without the implementer feeling"defensive." been acknowledgement of the need for individuals trained in fields
One way to look at the relative roles of the different entities involved other than just efficiency and/or with other skills,such as market
in preparing claimed and evaluated savings(as well as project analysis and public policy. People with skills such as interviewing and
savings)is to consider the roles as associated with oversight or survey data analysis are also needed.
administrator activities.These can be generally defined as follows: Many evaluators are members of industry professional organiza-
• Oversight activities:activities that are under the purview of tions or are Certified Measurement and Verification Professionals
the entity responsible for all energy efficiency programs and (CMVPs).99 The following are two of the professional organizations
the associated EM&V implemented by program administrators that energy evaluators participate in(and which will post evaluation
in the subject jurisdiction (e.g.,state or utility service terri- RFPs/RFQs):
tory).Oversight activities will usually include coordination with . Association of Energy Service Professionals,www.aesp.org
the government/regulatory authority.They may also include
• International Energy Program Evaluation Conference,
feedback or guidance to and from stakeholders(including
www.iepec.org.
administrators and implementers)about the evaluation plans
and implementation as well as the process of approving In addition,the California Measurement Advisory Council(CALMAC)
reported results. now offers a directory of evaluators,at www.calmac.org/
• Administrator activities:activities undertaken by the program contractorcontact.asp.
administrators during the process of developing,implementing,
and conducting M&V activities pertinent to their implementa- 8 Setting the Boundary and Granularity of
tion of energy efficiency programs.These M&V activities may Reported
rted Results
also be known as the primary evaluation activities. When evaluating energy,demand,and emissions savings,it is
important to properly define the project boundaries(i.e.,the equip-
As suggested in Section 8.2,it is in the evaluation framework that ment,systems,or facilities that will be included in the analyses).
these roles and responsibilities are sorted out.One example is how Ideally,all primary effects(the intended savings)and secondary
Maryland defined these roles in a simple matrix included in Table effects(unintended positive or negative effects—sometime called
8.1 at the end of this chapter.This is only an example,however,not interactive factors),100 and all direct(at the project site)and indirect
necessarily a recommended allocation of duties. (at other sites)effects will be taken into account. From a practical
point of view,with respect to energy and demand savings,this
In terms of selecting third-party independent evaluators,the usual translates into deciding whether savings will be evaluated for
approach is via a request for proposal(RFP)or request for qualifica- specific pieces of equipment(where the"boundary"may include,
tions(RFQ)process.Typical selection criteria include qualifications for example,just motor savings or light bulb savings),the end-use
and experience of particular staff assigned to the proposed work, system(such as the HVAC or the lighting system),whole facilities,or
resources that the bidding organization(or often,bidding team) an entire energy supply and distribution"system."
8-10 www.seeaction.energy.gov December 2012

From an electricity system point of view,the energy and demand then move the evaluation cycle and the program/portfolio cycle into
savings at the power plant producing the electricity will be greater better alignment within one or two years.Other evaluation activity
than the savings at the end-use(in the facility)due to transmission schedule considerations are the length of the portfolio cycle(one,
and distribution(T&D)losses.These average or marginal(which can two,or three years),whether to estimate persistence of savings(see
be much higher than the average)losses are on the order of 5%to Section 7.3),whether information is needed for a next portfolio cycle,
20%.101 Thus,the savings at the power plant busbar can be 120% and what information will be included in the evaluation reports.
of the end-use savings.Whether T&D losses will be included in the
boundary and reported as part of program impacts is something that In terms of reporting requirements,certainly there will be a need
can be defined in the evaluation framework. for final program and portfolio reports for each program year and/
or cycle.However,there will likely also be needs for interim reports
For avoided emissions calculations,the boundary assessment issues to track the performance of the programs being evaluated and the
are discussed in Chapter 6. Boundaries are also important for defin- evaluation itself.
ing other non-energy benefits(NEBs)as well.For example,if there is
job impact to be reviewed,it needs to be decided whether the jobs There are several considerations for setting up a reporting schedule:
are local,statewide, national,or international. • The timing for policy decisions and evaluation planning
The time granularity of evaluation analyses relates to whether 15-minute, • The desire to have early feedback for program implementers
hourly,monthly,seasonal,annual,and/or lifetime data collection and • Program lifecycle stage(evaluating a first-time program or a
savings reporting are required.The"granularity decision"is based on long-established program)
how the information from the evaluation is to be used.Annual savings • Evaluation data collection time lags
data are generally only useful for an overview of the program benefits. • Regulatory and/or management oversight requirements
More detailed data are usually required for cost-effectiveness analyses, Contract requirements for reporting savings for"pay for
demand forecasting,and resource planning.For avoided emissions, performance"programs
annual values are typical;however,for certain programs,such as smog
programs,there are specific seasons or time periods of interest. Timing requirements to use the evaluation results to update
energy and demand savings as well as measure life estimates
If demand savings are to be calculated,the choice of definition for specific energy efficiency measures in a TRM.
(e.g.,annual average,peak summer,coincident peak)is related to
time granularity of the evaluation results.Section 7.2 includes a A standard reporting plan would have final reports for each of
discussion of the different definitions and describes how this deci—
sion greatly influences the data collection requirements,and thus portfolio cycle evaluation reports,with interim reporting on the
the effort required to complete the evaluation. impact and portfolio cycle reports.Quarterly or biannual evaluation
status reports can also be helpful for keeping all stakeholders
8.3.4 Schedule for Evaluation and Reporting informed. In addition, providing online summary data to the public
The evaluation time frame has two components:the time period can be a valuable tool for keeping publicly funded efforts informed
over which the evaluation activities will take place and the reporting and perhaps for maintaining public support.
schedule.An ideal evaluation schedule begins before the start of
the program implementation(to collect any baseline data and set 8.3.5 How Are Evaluated Savings Estimates
Applied—Looking Back Or Going Forward?
up the overall evaluation infrastructure)and continues for some
time after the program is completed to analyze persistence of sav- Estimates of costs and savings from energy efficiency measures are
ings. However,the actual timing of the evaluation is influenced by typically made both prior to program implementation(i.e.,projected
several—often competing—considerations. savings)and post-program implementation(i.e.,evaluated and/or
claimed savings).And,as one would expect,evaluated estimates
The first consideration is when to start the evaluation efforts. of savings are considered a more accurate representation of actual
Programs and portfolios tend to get into a regular cycle;for example, savings than projected savings.An issue arises when deemed savings
the start of each program year may be on January 1,but the evalu- values are used to project and claim energy savings for an energy
ation process may not get started until the spring.One approach efficiency measure in a given program year(e.g.,based on per-unit
in this situation is to accept the late start of an evaluation for the savings values in a TRM approved for that program year),but an
current year and conduct a more limited effort than desired,but evaluation during the program year indicates the TRM per-unit
December 2012 www.seeaction.energy.gov 8-11

savings values are too high or too low for the subject measure. administrator goal achievement. However,once TRMs become
The question thus becomes"Should the deemed savings value be "stable"(i.e.,fewer and fewer changes are made)the differ-
adjusted retroactively to the current program year or only applied on ence should be minor,if there is any at all.
a going-forward basis?" Savings from custom projects or programs(where savings are
determined after project implementation using agreed-upon
Consider the following example: protocols)should use the evaluated values as the savings for all
• TRM per-unit savings values developed in 2011 indicate that purposes.
the savings from measures verified to have been installed in a
program in 2012 are 10,000 MWh. 8.3.6 Data Management Strategies
• However,an ex post(after the fact)evaluation indicates that Evaluations are based on data—often in very large amounts. Decision
the per-unit savings values in the TRM were overly optimistic, makers often do not need or want to see the detailed calculations or
and the actual program savings are 9,500 MWh based on the raw data inputs that drive evaluation results.But all parties need to
same number of units verified to have been installed. know that reported results are built on a foundation of sound data
• Assuming the verification is correct and all the measures inputs.Proper data management strategies allow administrators,
were installed,does the oversight body(e.g.,a regulatory
implementers,and evaluators—as well as oversight bodies—to
commission)credit the program with 10,000 MWh of savings delve into the underlying data,both to be able to review underlying
or only 9,500 MWh for the subject program year?
assumptions and to combine the data in new ways as they see fit for
current program reviews or future program developments.
There are equity issues associated with the above options for how
and when to apply updated TRM values.On one hand,the pro- Thus,while data management could easily be defined as"down in the
gram administrator/implementer relied on an approved value for weeds;'it can be a major attribute of a portfolio's implementation
budgeting and savings estimates;however,for resource planners and evaluation and a major cost.Thus,it deserves attention in the
and other stakeholders,what matters most is the most accurate evaluation framework.For stakeholders,the questions addressed in
indication of what occurred.While no perfect solution exists,this the framework tend to be associated with:"Will the data resources
is an issue that is best to decide upon for inclusion in an evaluation be fully used for current and future project development,such as
framework before it occurs during the portfolio cycle.One approach updating deemed savings values?""Will consumer confidentiality be
is to use this guidance: properly maintained?"and"What will be publicly accessible?"
• Per-unit cost and savings stipulated(deemed)values in a TRM
The fundamentals of good data management are the same across
should be based on the best available information at the time
industries.Within an energy efficiency portfolio,two areas of
these estimates and/or calculations are made,and they should
primary importance are the ability to compare results across time
be determined in a rigorous and transparent manner. It is (longitudinal analysis)and ability to compare results by factors such
recognized that TRM values are not static and will be updated
as program type or delivery mechanism.Some items are typically
from time to time.
specified as part of an evaluation plan:
• If new information indicates that per-unit cost and savings
• Data required
stipulated (deemed)values in a TRM should be updated,these
new values will be adopted for use in future program savings • Format of data to be provided by tracking systems;
projections,claims,and evaluations. compatibility and standardization
• Savings claimed in the current program year(based on deemed • Access to data and summaries
savings)for measures or programs already implemented before • Data confidentiality protection protocols
the TRM is updated are not adjusted retroactively up or down • Data quality assurance and control.
for purposes of defining program administrator or implementer
goal achievement. In terms of ensuring consistency of data reporting across programs,the
• In terms of using the evaluated results for load forecasting and following data components are recommended for standardization:
resource planning,estimates with the most up-to-date TRM Measure naming convention(i.e.,the same energy measures,
values are used,although this could result in two sets of results end-uses,and applications have the same name from program
being reported:one for forecasting/planning and one for to program)
8-12 www.seeaction.energy.gov December 2012

• Measure categorization (i.e., place measures into logical can and do arise,and it is best to define how they will be addressed
measure categories,subcategories,and technologies: before they occur.Most jurisdictions will have their own approaches,
Category=Lighting,Subcategory=Indoor Lighting, with mechanisms for regular discussions,regulatory hearings,media-
Technology=CFL) tions,arbitration,or other solutions.Even a few lines in an evaluation
• Sector and building type classification across programs framework document defining the steps for dispute resolution can
(e.g.,Sector=Commercial, Building Type=Education— eliminate a great deal of difficulty should a dispute arise.
Primary School). 8.4 PORTFOLIO CYCLE IMPACT EVALUATION
• Normalization units across programs(i.e., utilize normalization AND M&V PLAN OUTLINES
units as the basis of quantity or measure size,such Flow-GPM
for water flow measures or Ctrl-kW for occupancy sensors). The evaluation planning documents should clearly present the evalu-
• Program delivery methods across programs(e.g., program ation efforts and details of the actions to be undertaken during the
types are prescriptive rebates,direct install,and point of sale) evaluation activity,as well as consideration of regulatory(reporting)
requirements.A plan is a stand-alone decision document,meaning
• Indication of project dates(e.g.,date of installation). it must contain the information the evaluator and others need to
understand what is to be undertaken,why,when,and how.Plans
While there is not currently a single reporting standard for energy are also important historical documents since it is not unusual for
efficiency programs,national,regional,and international reporting programs with long lifecycles to undergo staff changes over the
specifications are under development.In the meantime,three data- course of the program.
base projects that demonstrate good data management practices are
as follows: The following subsections outline the contents of a portfolio cycle
• Standardized Project Tracking(SPT)database in California. impact evaluation plan and M&V plan.The M&V plan is included
California has long maintained various databases related to because it is a common approach for calculating gross energy savings.
energy use and energy efficiency. In recent years,the California Following the M&V plan outline is an evaluation planning checklist.
Public Utilities Commission has worked closely with the
program administrators to create standard claims-reporting 8.4.1 Portfolio Cycle Impact Evaluation Plan and
templates. Recent versions of the SPT allow all stakeholders Report Outlines
in California to review program savings and costs at the The following is an outline that can be used to produce an impact
measure, program,and portfolio levels.See the websites at evaluation plan.
http://open-emv.com/and www.cpuc.ca.gov/PUC/energy/ Part A.Program Background
Energy+Efficiency. 1. Short description of the program(s)being evaluated (e.g.,the
• The Regional Energy Efficiency Database(REED). Developed market,program delivery approach,technologies,budget,
by the Regional EM&V Forum, REED has commitments from objectives)
ten jurisdictions for a common reporting system of energy 2. Presentation of how the program will achieve its objectives,the
efficiency savings and associated impacts to support a range of program theory
energy and environmental policies.See the website at 3. List of the technologies offered by the program
http://neep.org/emv-forum. 4. Program schedule
• The Northwest Regional Technical Forum(RTF).The RTF is an 5. Numerical energy and non-energy savings projections.
advisory committee established in 1999 to develop standards
to verify and evaluate conservation savings.See the website at Part B.Evaluation Overview
www.nwcouncil.org/energy/rtf/Default.htm. 1. List of evaluation objectives and how they support program goals
8.3.7 Addressing Disputes 2. List of which metrics will be reported (e.g.,annual MWh,
Disputes can arise from the process used to develop impact monthly peak kW,annual therms,annual COz)
evaluation results and/or the results themselves. Disputes are best 3. Description of verification activities
addressed before they arise through understood agreements on 4. Version of the TRM to be used and/or any TRM development/
how the evaluations will be conducted(i.e.,defined in an evaluation review activities
framework and through good communication).However,disputes
December 2012 www.seeaction.energy.gov 8-13

5. Gross and net impact evaluation approaches selected for 5. An activities timeline with project deliverable dates,
determining energy(and demand)savings including reporting activity and key milestones,including
6. Methodology for calculating non-energy benefits such as communications with administrator/implementer
avoided emissions,as appropriate 6. Detailed budget and schedule
7. List of primary factors that will be considered in analysis of 7. Evaluation team—information concerning the independence of
gross and net savings(e.g.,weather,occupancy,free riders, the evaluator.
spillover)as well as list of major assumptions
8. Description of how program impact results will be combined to The final product or output of an evaluation is an evaluated savings
report portfolio impacts,addressing the need for adjustments report.The following is a sample report outline:
such as accounting for program overlap or other factors • List of Figures and Tables
9. Expectations for overall certainty of savings estimates
• Acronyms
• Abstract
10. Assumptions concerning availability of data and other • Acknowledgments
information provided by the administrator/implementer;
relative roles of evaluator and administrator/implementer 1. Executive Summary(Include highlights of key recommended
11. Budget and schedule summary improvements to the program,if relevant)
12. Listing of evaluators(if known)or evaluator selection method. 2. Introduction
• Program Overview(e.g.,program description,objectives)
Part C.Detailed Evaluation Approach,Scope,Budget, • Evaluation Objectives and Methods
Schedule,and Staffing • Report Structure
(This is the detailed presentation of evaluation activities to be 3. Study Methodology
undertaken,including the evaluation approach to be used.) • Data Collection Approach(es)
1. Gross impact savings analysis—a description of the data • Analysis Methods
collection and analysis activities and approaches(if an M&V • Limitations,Caveats
evaluation approach is selected, identify the IPMVP option to 4. Key Evaluation Results with metrics and realization rates
be used) (answers for all of the questions specified for the evaluation.
2. Net impact savings analysis—a description of how spillover, This could include several sections on findings. Findings could
free ridership,and other effects will be addressed in the be presented for each method used, by program components
evaluation activities and in the data analysis(as appropriate) covered,by market segments covered,and so forth,followed
by a section on integrated findings or organized and presented
3. Data collection,handling,and sampling: by the different observed effects or type of results.)
a. Measurement collection techniques 5. Synthesis of analysis and findings as well as implications of
b. Sampling approach and sample selection methods for findings
each evaluation activity that includes sampling efforts 6. Recommendations(should include clear,actionable,and
(as appropriate) prioritized recommendations that are supported by the analysis)
c. How the comparison group or non-participant information 7. Summary and Conclusions
will be used in the evaluation(s)and in the analysis
8. Appendices(examples listed below):
(as appropriate)
d. Data handling and data analysis approach to be used to • Recommended improvements to the evaluation process,
address the researchable issues including any lessons learned for future evaluation studies
4. Uncertainty of results—presentation and discussion of the • Detailed documentation of the research design and
threats to validity, potential biases, methods used to minimize assumptions,data collection methods,evaluation analysis
bias,and level of precision and confidence associated with methodology,references,and results tables
the sample selection methods and the evaluation approaches; • Survey or interview instrument,coding scheme,and
quality control information should also be included here compiled results tables and data
8-14 www.seeaction.energy.gov December 2012

• Sources and quality(caveats on data)of primary and The M&V plan should include an overall project description,facility
secondary information equipment inventories,descriptions of the proposed measures,energy-
• Details on quantitative data analysis:analytical framework, savings estimates,a budget,and proposed project implementation and
modeling approach,and statistical results M&V schedules.A project-specific M&V plan should also demonstrate
• Qualifications to results that any metering and analysis will be done consistently,logically,and
• Possible sources of overestimation and underestimation with a level of accuracy acceptable to all parties.
• Analysis of reliability of energy savings estimates,treatment
The following is a sample M&V plan outline:
of issues that threaten reliability of results(e.g.,double
counting, use of savings factors,and synergistic effects) 1. Description of project, measures to be installed,and
How attribution was addressed(for net savings impact project objectives
• evaluation) 2. elected IPMVP option and measurement boundary
• Other assumptions and justifications. 3. Description of base year conditions,data collection,
and analyses
8.4.2 Project-Specific M&V Plan and Report Outlines 4. Identification of any changes to base year conditions and
If the M&V gross impact evaluation approach is selected,an M&V how they will be accounted for in the analyses
plan needs to be prepared for each project selected for analysis.This 5. Description of reporting period conditions,data collection,
section discusses the M&V planning process for individual projects and analyses
and presents an M&V plan outline and report outline.
6. Basis for adjustments that may be made to any measurements
M&V activities involve the following steps: and how this will be done
1. Selecting one of the four IPMVP options for the project.The 7• Specification of exact analysis procedures
options define general approaches to documenting savings. 8. Metering schedule and equipment specifications
2. Preparing a project-specific M&V plan that outlines the details 9. Description of expected accuracy and how it will be determined
of what will be done to document savings. 10. Description of quality assurance procedures
3. Defining the pre-installation baseline,including equipment 11. Description of budget and schedule
and systems, baseline energy use,and factors that influence 12. Description of who will conduct M&V.
baseline energy use.
4. Defining the reporting period situation,including equipment The tables at the end of this chapter summarize what could be
and systems, post-installation energy use,and factors that contained in the M&V plans.Table 8.2 lists general requirements
influence post-installation energy use.Site surveys;spot,short- for an overall M&V plan.Table 8.3 lists requirements that could be
term,or long-term metering;and/or analysis of billing data can addressed in the M&V plan for each measure(e.g.,building lighting
also be used for the reporting period assessment. retrofit,building air-conditioning retrofit,control system upgrade)
5. Conducting periodic(typically annual) M&V activities to verify that is included in the project being evaluated. More information on
the continued operation of the installed equipment or system, the contents of an M&V Plan can be found in the IPMVP
determining current year savings,identifying factors that may (www•evo-world.org).
adversely affect savings in the future,and estimating savings 8.4.3 Checklist of Planning Decisions for an
for subsequent years. Impact Evaluation
A project-specific M&V plan describes in reasonable detail what Table 8.4 presents a checklist for preparing an impact evaluation
will be done to document project savings. It can be a plan for each plan.The list is organized around the decisions associated with the
energy efficiency measure included in the project(e.g.,when a gross savings calculation,net savings calculation,calculation
retrofit isolation approach is used).Or,it can cover the entire project of avoided emissions,and generic issues.
(e.g.,when the whole-facility analysis approach is used). In either
case,the M&V plan will consider the type of energy efficiency
measures involved and the desired level of savings certainty.
December 2012 www.seeaction.energy.gov 8-15

TABLE 8.1: Maryland Public Service Commission EM&V, Data Tracking, and Reporting Act Roles and
Responsibilities for Utilities and Statewide Evaluation Consultants
AND/ORUTILITIES PSC STATEWIDE
TASK DELIVERABLE (OR THEIR EVALUATION
CONSULTANT) •
STATEWIDE STUDIES
Prepare statewide baseline study X102
Prepare additional statewide market assessments(e.g.,market
X
impact studies)and updates(bi-or tri-annual)
Prepare statewide technical reference manual(TRM) X103
(and annual or bi-annual updates)
Review TRM and updates of TRM X
Approve TRM(and review statewide market assessments) X
PLANNING AND MANAGEMENT
Prepare Master Evaluation Schedule for utility and statewide
evaluation contractor activities;include overview of reporting
X X X
schedule with annual and tri-annual portfolio reporting,as well as
semi-annual interim reports and/or presentations
Develop utility(statewide and individual utility)impact and process
evaluation plans;including gross and net energy and demand
(including peak demand from distributed resources)savings, X
cost-effectiveness analyses,database and reporting protocols,
survey templates,and schedules
Review utility evaluation plans X
Develop plan for due diligence(quality assurance/quality control,
QA/QC)of utility impact results(energy and demand savings,cost
X
effectiveness),including verification approach(with sampling plan)
and schedules for review of utility submittals and reporting to PSC
8-16 www.seeaction.energy.gov December 2012

UTILITIES PSC
• ' EVALUATION• •TASK • ' • PSC
• •
PLANNING AND MANAGEMENT(CONTINUED)
Develop plan for determining and reporting additional outcomes
(e.g.,system reliability,;T&D and generation needs impacts;emissions
X X
avoidance;price mitigation;jobs impacts;effects on ratepayers,
especially low-income;coordination with federal stimulus funding)
Review and approve utility and PSC evaluation consultant plans X
Coordinate all utility evaluation efforts X
Coordinate statewide due-diligence(QA/QC)impact evaluation/ X
verification efforts
PROCESS EVALUATION
Prepare program process evaluations X
Conduct(independent)customer and trade-ally satisfaction surveys
X
and reports
Review customer and trade-ally satisfaction survey results X x
DATABASES
Prepare a data reporting,interface,and database plan that includes
coordination between the utilities'and PSC's statewide evaluation X X
contractor database(s)and utility databases
Design,implement,and maintain utility primary program
management and tracking database(s)with project and program
X
data(includes individual utility databases and a statewide database
operated by utilities)
Design,implement,and maintain statewide data management and
quality control database of information"uploaded"from utility
database(s)and used for(1)obtaining and managing data for
X
due-diligence activities,and(2)establishing a public web-accessible
database and reporting system with aggregated,higher-level informa-
tion on program impacts(e.g.,statewide energy and demand savings)
Review and approve statewide database and reporting plan I X
December 2012 www.seeaction.energy.gov 8-17

AND/ORUTILITIES PSC STATEWIDE
TASK DELIVERABLE (OR THEIR EVALUATION
CONSULTANT) •
PRIMARY DATA COLLECTION AND IMPACT ANALYSES: ENERGY EFFICIENCY(EE)AND DEMAND RESPONSE(DR)
rPROGRAMS
Prepare ex ante(preliminary)savings estimates X
Conduct primary data collection and site baseline and ex post(after the
X
fact)verifications for energy efficiency and DR projects
Prepare persistence of savings analysis:Conduct primary
X
data collection
Prepare analyses and documentation of project,program,and
X
portfolio gross and net energy and demand savings,cost effectiveness
INDEPENDENT DATA COLLECTION AND IMPACT ANALYSES
Conduct quality control and due diligence of utility analyses and
documentation of project,program,and portfolio gross and net energy
and demand savings and cost effectiveness;inspect sample of project sites X
and review primary data and analyses,prepare verified achieved versus
claimed savings and cost-effectiveness report per reporting schedule
I
OTHER OUTCOME ANALYSES
Prepare additional efficiency and DR program/portfolio outcome
reports:System reliability,T&D and generation needs impacts;emissions
avoidance; price mitigation;jobs creation;impacts on low-income X
ratepayers;and leveraging of federal stimulus funding such as American
Recovery and Reinvestment Act(ARRA)funds
CREPORTING
Prepare utility interim semi-annual and final annual(and tri-annual)
reports of energy efficiency program and portfolio net and gross impacts X
and cost-effectiveness evaluation results
Prepare semi-annual(interim)and annual(final)reports of verified
achieved utility program and portfolio results for energy and demand X
savings and cost effectiveness.
Prepare annual report on additional Empower MD outcome results(e.g.,
X
avoided emissions,reliability and job impacts)
Review utility and PSC consultant semi-annual reports; review and approve
X
utility and PSC consultant annual(and tri-annual)reports
8-18 www.seeaction.energy.gov December 2012

UTILITIES PSC STATEWIDE
TASK • - • - THEIR EVALUATION
CONSULTANT) •
BEST PRACTICES
Participate in quarterly(or semi-annual)impact evaluation process review
X X X
and improvement meetings
Prepare best practices recommendations for improvements to
X X
evaluation processes
Review and approve best practices recommendations for program
X
modifications and improvements
OTHER
Prepare materials and reports in support of PSC analysis of
efficiency programs X X
Organize and conduct periodic statewide public workshops on evaluation
X X
results of energy efficiency programs
Source:Consensus Report on the EmPower Maryland EM&V Process.(June 24,2009). .
December 2012 www.seeaction.energy.gov 8-19

TABLE 8.2: Energy Efficiency M&V Plan Contents: General Components
. • M&V PLAN COMPONENTS
Project goals and objectives
Project Description Site characteristics and data available(e.g.,whole building and end-use metering)
Measure descriptions that include how savings will be achieved
Estimated savings by measure
Project Savings and Costs
Estimated M&V cost by measure
Project installation schedule
Schedules
M&V activities
Raw and compiled data formats
Reporting M&V report contents
Reporting interval
Confidence and precision of results
M&V Approach M&V option(s)used
Person(s)responsible for M&V activities
8-20 www.seeaction.energy.gov December 2012

TABLE 8.3: Energy Efficiency Project-Specific and Measure-Specific M&V Plan Contents
M&V PLAN COMPONENTS OMMMMMMI
Data requirements kW,operating hours,temperature
Basis of stipulated values Lighting operating hours equal 4,000/year based on
metered XYZ building
Analysis Method kWh savings(year)=[(kW/Flxture(baseline)x Quantitylbaseline)]
Savings calculated equations —(kW/Fixture(post)x Quantity(post)]x Operating Hours per
yea r(post)
Regression expressions Three parameter change-point cooling model
Computer simulation models DOE-2 simulation model
Metering protocols ASHRAE Guideline 14-2002 pump multiple point test
throughout short-term monitoring
Equipment ABC Watt Hour Meter
Equipment calibration protocols National Institute of Science and Technology protocols
Metering and Monitoring
Metered data Flow rate,RMS power
Sample size 25 lighting circuits out of 350
Sampling accuracy 90%confidence/10%precision
Metering duration and interval 2 weeks/15-minute data
Performance factors Boiler efficiency
Operating factors Load,operating hours
Baseline Determination
Existing service quality Indoor temperature set points
Minimum performance standards State energy code
Party responsible for developing
Smith Engineers,hired by sponsor
adjustments
Savings Adjustments
Savings adjustment approach Baseline adjusted for reported period weather and
building occupancy levels
December 2012 www.seeaction.energy.gov 8-21

TABLE 8.4: Evaluation Planning Checklist
CHECKLIST FOR GROSS SAVINGS DETERMINATION
SAVINGS TO BE REPORTED
Energy savings(annual,seasonal,monthly,hourly,other)
Demand savings(peak,coincident,average,other)
SELECTED GROSS ENERGY SAVINGS CALCULATION APPROACH
Measurement and verification approach
Deemed savings approach
Large-scale billing analysis approach
Quality assurance approach
L MEASUREMENT AND VERIFICATION APPROACH
IPMVP Option A,B,C,or D
DEEMED SAVINGS APPROACH
Source of deemed savings identified and verified
LARGE-SCALE BILLING ANALYSIS APPROACH
Randomized controlled trials
Quasi-experimental method
SAMPLE SIZE CRITERIA SELECTED
LIFETIME ENERGY AND DEMAND SAVINGS FACTORS TO BE EVALUATED
Degradation
LRebound
Other
8-22 www.seeaction.energy.gov December 2012

. CHECKLIST FOR NET SAVINGS DETERMINATION
NET SAVINGS FACTORS TO BE EVALUATED
Free riders
Spillover effects
Other market effects
Other(s)
NET SAVINGS CALCULATION APPROACH SELECTED
Self-reporting surveys
Enhanced self-reporting surveys
Large-scale consumption data analysis approaches(randomized controlled trial methods and
quasi-experimental methods)
Stipulated net-to-gross ratio
Cross-sectional studies
Top-down evaluations(or macro-economic models)
CHECKLIST FOR AVOIDED EMISSIONSCALCULATIONS
ELECTRICITY EFFICIENCY SAVINGS—GRID-CONNECTED
LLoad emission rates
Regional marginal baseload emission rates(using capacity factors or equivalent)
Regional historical hourly emission rates
Energy scenario modeling
NATURAL GAS, FUEL OIL,AND NON-GRID-CONNECTED ELECTRIC GENERATING UNITS
Default emission factor
Source testing
December 2012 www.seeaction.energy.gov 8-23

EVALUATIONGENERIC • •
OVERALL GOALS
L
Does the evaluation address the key policy,regulatory,and oversight needs for evaluation information?
Will the program's success in meeting energy,demand,and emissions goals be quantifiably evaluated in the
same manner as they are defined for the program?
Does the evaluation plan represent a reasonable approach to addressing the information needs?
Are there missing opportunities associated with the evaluation approach that should be added or considered?
Are any additional non-energy benefits being evaluated?
Does the impact evaluation provide the data needed to inform other evaluations that may be performed,
particularly cost-effectiveness analyses?
Has a balance been reached between evaluation costs,uncertainty of results,and value of evaluation results?
UNCERTAINTY OF EVALUATION RESULTS
Can the confidence and precision of the evaluation results be quantified?If so,how?
Are there key threats to the validity of the conclusions?Are they being minimized given budget constraints
and study trade-offs?Will they be documented and analyzed?
Is the evaluation capable of providing reliable conclusions on energy and other impacts?
BUDGET,TIMING,AND RESOURCES
Does the evaluation take advantage of previous evaluations and/or concurrent ones for other programs?
Does the cost of the study match the methods and approaches planned?
LDo the scheduled start and end times of the evaluation match the need for adequate
Are adequate human resources identified?
Does the evaluation rely on data and project access that are reasonably available?
REPORTING
Are the time frames and scopes of evaluation reported defined?
Do the data collection,analysis,and quality control match the reporting needs?
Are the persistence of savings and avoided emissions being evaluated?
LHave measurements and impacts(emissions)boundaries been properly set?
8-24 www.seeaction.energy.gov December 2012

EVALUATIONGENERIC • . •
SAMPLING AND ACCURACY
L
Is the sampling plan representative of the population served?
Is the sampling plan able to support the evaluation policy objectives?
LAre there threats to the validity of the evaluation results addressed in the sample design?
December 2012 www.seeaction.energy.gov 8-25

Chapter 8: Notes
96 See discussion of theory-based evaluation in Section 8.3.1.
97 Cross-border trading means that energy savings obtained in one
jurisdiction cannot be used to meet energy efficiency requirements in
another jurisdiction.Should a national energy efficiency resource
portfolio be established at some point,then consistent national EM&V
standards and deemed savings values/calculations may become
required. For more information on EM&V requirements that may be
associated with a national energy efficiency or clean-resource
standard,see State and Local Energy Efficiency Action Network.(April
2011).National Energy Efficiency Evaluation, Measurement, and
Verification(EM&V)Standard:Scoping Study of Issues and
Implementation Requirements. Prepared by Schiller,S.R.;Goldman,
C.A.;Galawish, E.; LBNL Environmental Energy Technologies Division.
wwwl.eere.energy.gov/seeaction/pdfs/emvstandard_scopi ngstudy.pdf.
98 Kushler, M.; Nowak,S.;Witte, P.(February 2012).A National
Survey of State Policies and Practices for the Evaluation of Ratepayer-
Funded Energy Efficiency Programs.American Council for an Energy-
Efficient Economy(ACEEE). Report Number U122.www.aceee.org/
resea rch-report/u 122.
99 See the Efficiency Valuation Organization(EVO):www.evo-world.org.
100 A very common interactive factor is the relationship between(1)
reductions in heat generated by retrofitted lighting systems in a building
as they consume less electricity and(2)the possible requirements for
increased space heating and/or reduced space cooling.
101 Lazar,J.; Baldwin,X.(August 2011).Valuing the Contribution of
Energy Efficiency to Avoided Marginal Line Losses and Reserve
Requirements. Regulatory Assistance Project(RAP).www.raponline.org/
document/down load/id/4537.
102 As of now,the baseline study contractor is separate from statewide
contractors hired by the PSC for other activities listed in this table.
103 May be done in conjunction with Northeast Energy Efficiency
Partnerships(NEEP)effort.
8-26 www.seeaction.energy.gov December 2012

Appendix A: Glossary
This glossary defines and explains terms used in this guide and in
the evaluation,measurement,and verification(EM&V)of energy
efficiency programs.Included are terms commonly used in the design Savings,or more accurately stated"savings estimates,"from
and implementation of EM&V,terms often found in EM&V reports, energy-efficiency measures,projects,programs,and portfolios
and terms associated with efficiency measures,programs,and are reported at various times in the lifecycle of the efficiency
program strategies.Some of the definitions in this glossary reflect activity and with varying degrees of certainty.The two major
usage in the specific context of energy efficiency program administra- points when they are reported are prior to and after the
tors and implementers with an oversight(regulatory)body;thus,they implementation of the activity.Different jurisdictions have
may not be applicable to non-regulatory contexts such as in commer- different names for savings reports,what they contain,and
cial agreements between energy services companies and their clients. whether and what adjustments or evaluation activities take
place between pre-implementation and post-implementation.
References used for preparation of this glossary are given at the For example,prior to implementation,savings can be called
end of this appendix.Glossaries tend to be living documents. For Ex Ante Savings or Projected Savings.After implementation,
example,the well-regarded "EM&V Forum Glossary of Terms,"a there are even more variations in names and content,with
project of the Regional Evaluation, Measurement and Verification titles including ex post savings,adjusted savings,evaluated
Forum is regularly updated as a result of reviews by forum members savings,and tracking estimates.Examples of differences in
and outside experts.104 these"after implementation"reports can include whether the
results have been corrected for data errors,whether they have
been verified(e.g.,confirmed actual number of installations
Accuracy:A concept that refers to the relationship between the true versus planning projections),and/or whether a third-party
value of a variable and an estimate of the value.The term can also evaluator has generated or reviewed the reported values.
be used in reference to a model or a set of measured data,or to Savings can also be indicated as first-year,annual demand
describe a measuring instrument's capability. or energy savings,and/or lifetime energy or demand savings
values.They also can be indicated as gross savings and/or net
Additionality:A criterion applied to,for example,greenhouse gas savings values.In this glossary,several savings reporting
(GHG)projects,stipulating that project-based GHG reductions classifications are defined.Ideally,these terms and meth-
should only be quantified if the project activity"would not have ods should be applied;however,it is critical that whenever
happened anyway;"in other words,that the project activity(or the savings are reported,the basis for the values indicated
same technologies or practices it employs)would not have been
be made clear,rather than just relying on these common
implemented in its baseline scenario and/or that project activity
classifications,since their definition(and titles)—as well
emissions are lower than baseline emissions.
as how values are determined for each classification—can
Adjusted Savings:See claimed savings and evaluated savings. vary from jurisdiction to jurisdiction.
Administrator:An entity selected by a regulatory or other
government organization to contract for and administer an energy
efficiency portfolio within a specific geographic region and/or
sufficient allowances to cover their pollutant emissions during
that period.Allowances are often confused with credits earned in
market.Typical administrators are utilities selected by a public
the context of project-based or offset programs,in which sources
service commission or a nonprofit or state government agency,as
trade with other facilities to attain compliance with a conventional
determined by legislation.
regulatory requirement.Cap-and-trade program basics are discussed
The basic tradable commodity within a pollutant at www.epa.gov/capandtrade/index.html.
emissions trading systems.Allowances grant their holder the right to
Annual Demand Savings:The maximum reduction in electric or gas
emit a specific quantity of pollution once(e.g.,one ton of COz(eq)).
The total quantity of allowances issued by regulators dictates the demand associated with energy efficiency activities in a given year
within a defined boundary.More specific definitions of demand and
total quantity of emissions possible under the system.At the end
of each compliance period,each regulated entity must surrender thus demand savings occur within specific regulatory jurisdictions or
electric control areas.
December 2012 www.seeaction.energy.gov A-1

Annual Energy Savings:The reduction in electricity use(kilowatt- Baseline Data:The baseline conditions of the facilities,market
hours)or in fossil fuel use(in thermal units)associated with energy segment,generating equipment,or other area of focus of the subject
efficiency activities in a given year within a defined boundary. efficiency activity.
ASHRAE Guideline 14:American Society of Heating, Refrigerating Baseline Efficiency:The energy use of the baseline equipment,
and Air-Conditioning Engineers(ASHRAE)Guideline 14,2002 process,or standard that is being replaced by a more efficient
Measurement of Energy and Demand Savings(www.ashrae.org). approach to providing the same energy service(a subset of
baseline). It is used to determine the energy savings obtained by the
The boundary within which all the primary more efficient approach.
effects and significant secondary effects associated with a project
are evaluated. Baseline Period:The period of time selected as representative of
facility operations before the energy efficiency activity takes place.
Attribution:Ascribing or establishing a causal relationship between
action(s)taken by an entity and an outcome. For efficiency program Baseline Scenario:A hypothetical description of what would have
evaluation,this is associated with the difference between net and most likely occurred without implementation of the subject project
gross savings. For example,an impact evaluation indicates that 30% or program.
of the gross energy savings associated with a ceiling fan incentive
program could be attributed to the ENERGY STAR'labeling program Behavior Programs:Energy efficiency programs that use strategies
and not the incentive program. intended to affect consumer energy use behaviors in order to
achieve energy and/or energy demand savings. Programs typically
Avoided Costs:The forecasted economic benefits of energy savings. include outreach,education,competition,rewards, benchmarking,
In the context of energy efficiency,these are the costs that are and/or feedback elements.Such programs may rely on changes
avoided by the implementation of an energy efficiency activity.Such to consumers' habitual behaviors(e.g.,turning off lights)or"one-
costs are used in benefit-cost analyses of energy efficiency activities. time"behaviors(e.g.,changing thermostat settings). In addition,
Because efficiency activity reduces the need for electric generation, these programs may target purchasing behaviors(e.g.,purchases
these costs include those associated with the cost of electric of energy-efficient products or services),often in combination with
generation,transmission,distribution,and reliability.Typically,costs other programs(e.g., rebate programs or direct install programs).
associated with avoided energy and capacity are calculated.Other These programs are also distinguished by normally being evaluated
costs avoided by the efficiency activity can also be included,among using large-scale data analysis approaches involving experimental
them the value of avoided emissions not already embedded in the or quasi-experimental methods,versus deemed savings or
generation cost,impact of the demand reduction on the overall measurement and verification approaches.
market price for electricity,and avoided fuel or water. For natural gas
efficiency programs,avoided costs can include components of the Benchmarking:A process that compares the energy,emissions,
production,transportation,storage,and service that are variable to and other resource-related conditions of a facility against industry
the amount of natural gas delivered to customers. best practices or other benchmarks such as average per square foot
energy consumption of similar buildings in the same city.
Baseline:Conditions,including energy consumption and demand,
which would have occurred without implementation of the subject Benefit-Cost Ratio:The mathematical relationship between the
energy efficiency activity.Baseline conditions are sometimes referred benefits and costs associated with the implementation of energy
to as"business-as-usual"conditions and are used to calculate efficiency measures,programs,practices,or emission reductions.
project-and program-related savings. Baselines can also include The benefits and costs are typically expressed in dollars.
definition of non-energy metrics that are being evaluated,such as
air emissions and jobs. Benefit-Cost Test:The methodology used to compare the benefits
of an investment with the costs;also called cost-effectiveness test.
Baseline Adjustments:For measurement and verification analyses, Five key benefit-cost tests have,with minor updates,been used
factors that modify baseline energy or demand values to account for for more than 20 years as the principal approaches for energy
independent variable values(conditions)in the reporting period. efficiency program evaluation.These five cost-effectiveness tests are
A-2 www.seeaction.energy.gov December 2012

the participant cost test,the program administrator cost test the Calibration: In economic,planning,or engineering modeling,the
ratepayer impact measure test the total resource cost test, and the process of adjusting the components of the model to reflect reality
societal cost test. as closely as possible,in order to prepare for the model's use in
future applications.The term also applies to the process whereby
Bias:The extent to which a measurement or a sampling or analytic metering and measurement equipment is periodically adjusted to
method systematically underestimates or overestimates a value. maintain industry measurement standards.
Some examples of types of bias include engineering model bias;
meter bias;sensor placement bias;inadequate or inappropriate California Measurement Advisory Council(CALMAQ An informal
estimates of what would have happened absent a program or committee comprised of representatives of the California utilities,
measure installation;a sample that is unrepresentative of a state agencies,and other interested parties.CALMAC provides
population;and selection of other variables in an analysis that are a forum for the development,implementation,presentation,
too correlated with the savings variable(or each other)in explaining discussion,and review of regional and statewide market assessment
the dependent variable(such as consumption). and evaluation studies for California energy efficiency programs
conducted by member organizations(www.calmac.org).
Billing Analysis:A term used to define either(1)a specific
measurement and verification(M&V)approach used to estimate Claimed Savings:Values reported by an implementer or
project savings or(2)any analytic methodology used to determine administrator,using their own staff and/or an evaluation consulting
project or program energy savings based on the use of the energy firm,after the subject energy efficiency activities have been
consumption data contained in consumer billing data.It compares completed;also called tracking estimates,reported savings,or in
billing data from program participant(s)over a period of time before some cases,ex post savings(although ex post usually applies to
the energy-efficient measures are installed at customer site(s)to evaluated savings[see evaluated savings]).
billing data for a comparable period of time afterward.If used to
describe an M&V approach,it is equivalent to IPMVP Option C,Whole As with projected savings estimates,these values may use results
Building Analysis.If used to describe an evaluation approach,it is of prior evaluations and/or values in a technical reference manual.
comparable to the large-scale data analysis approach. However,they may be adjusted from projected savings estimates by
correcting for any known data errors and actual installation rates and
Billing Data:Data obtained from the electric or gas meter that may also be adjusted with revised values for factors such as per-unit
is used to bill the customer for energy used in a particular billing savings values,operating hours,and savings persistence rates.
period.In an evaluation context,billing data also refers to the Claimed savings can be indicated as first-year,annual,and/or lifetime
customer billing records over time.Those records are used to energy or demand savings values,and can indicated as gross savings
conduct analyses of energy use before and after implementation and/or net savings values.
of energy efficiency measures.
Co-Benefits:The impacts of an energy efficiency program other than
Building Commissioning:See commissioning. the direct purpose(i.e.,energy and demand savings)for which it was
designed.See non-energy benefits.
Building Energy Simulation Model-Computer models based on
physical engineering principals and/or standards used to estimate Coefficient of Variation(CV The mean(average)of a sample,
energy use and/or savings.These models usually incorporate divided by its standard error.
site-specific data on customers and physical systems,such as
square footage,weather,surface orientations,elevations,space Coincident Demand:The demand of a device,circuit,or building
volumes,construction materials,equipment use,lighting,and that occurs at the same time as the peak demand of a utility's system
building occupancy.Building simulation models can usually account load or at the same time as some other peak of interest,such as
for interactive effects between end uses(e.g.,lighting and HVAC), building or facility peak demand.The peak of interest should be
part-load efficiencies,and changes in external and internal heat specified(e.g.,"demand coincident with the utility system peak").
gains/losses.Examples of building simulation models include DOE-2, The following are examples of peak demand:
EnergyPlus,and Carrier HAP. • Demand coincident with utility system peak load
December 2012 www.seeaction.energy.gov A-3

• Demand coincident with independent system operator/regional variable are associated with high values of another variable for the
transmission organization summer or winter peak or according same participant. For example,facility size and energy consumption
to performance hours defined by wholesale capacity markets usually have a high positive correlation.
• Demand coincident with high electricity demand days.
Cost-Benefit and Cost-Effectiveness Analysis:Analysis that compares
Commissioning:Often abbreviated as"Cx,"a systematic quality the benefits associated with a program or measure's outputs or
assurance process associated with new construction that spans outcomes with the costs(resources expended)to produce them.
the entire design and construction process,helping ensure that a Cost-benefit analysis is typically conducted to determine the
new building's performance meets owner expectations;sometimes relationship of the program's benefits and costs,as a ratio.Cost-
referred to as building commissioning.Commissioning ensures effectiveness analysis is generally undertaken to compare one
that the new building operates as the owner intended and that program or program approach to other approaches,or options for
building staff are prepared to operate and maintain its systems and the use of funds,to determine the relationship among the options.
equipment. The terms are often interchanged in evaluation discussions.See
benefit-cost test.
Common Practice:The predominant technology(ies)implemented
or practice(s)undertaken in a particular region or sector.Common Cost Effectiveness:An indicator of the relative performance or
practices can be used to define a baseline. economic attractiveness of any energy efficiency investment or
practice. In the energy efficiency field,the present value of the
Comparison Group:See control group. estimated benefits produced by an energy efficiency program is
compared to the estimated total costs to determine if the proposed
Conditional Savings Analysis(CSA):A type of analysis in which investment or measure is desirable from a variety of perspectives
change in consumption is modeled using regression analysis against (e.g.,whether the estimated benefits exceed the estimated costs
presence or absence of energy efficiency measures. from a societal perspective).See benefit-cost ratio.
Confidence:An indication of how close,expressed as a probability, Cumulative Energy Savings:The summation of energy savings(e.g.,
the true value of the quantity in question is within a specified megawatt-hours,therms)from multiple projects or programs over a
distance to the estimate of the value.Confidence is the likelihood specified number of years,taking into account the time of measure
that the evaluation has determined the true value of a variable installation in the first year,annual energy savings for subsequent
within a certain estimated range. For example,a program that years,and the average life of the installed measures.
produces an estimate of 2%energy savings,with a 95%confidence
interval of(1%,3%)means that there is a 95%probability that the Custom Program:An energy efficiency program intended to provide
true program energy savings is between 1%and 3%.A smaller efficiency solutions to unique situations not amenable to common
confidence interval implies that the estimate is more precise(e.g., or prescriptive solutions. Each custom project is examined for its
a 2%energy savings estimate with a confidence interval of[1.5%, individual characteristics,savings opportunities,efficiency solutions,
2.5%]is more precise than a 2%energy savings estimate with a and often,customer incentives.
confidence interval of[1%,3%]).See precision.
Database for Energy-Efficient Resources(DEER):A California
Control Group:A group of consumers who did not participate in the database designed to provide publicly available estimates of energy
evaluated program during the program year and who share as many and peak demand savings values,measure costs,and effective useful
characteristics as possible with the treatment(participant)group; life(www.deeresources.com).
also called comparison group.The comparison group is used to
isolate program effects from other factors that affect energy use. Deemed Savings Calculation:An agreed-to(stipulated)engineering
algorithm(s)used to calculate the energy and/or demand savings
Cooling Degree Days:The cumulative number of degrees in a month associated with an installed energy efficiency measure.These
or year by which the mean temperature is above a set temperature, calculations are developed from common practice that is widely
usually 18.3°C/65°F.See degree days. considered acceptable for the subject measure and its specific
application.It may include stipulated assumptions for one or
Correlation For a set of observations,such as for participants in an
energy efficiency program,the extent to which high values for one
A-4 www.seeaction.energy.gov December 2012

more parameters in the algorithm,but typically it requires users Demand-Side Management(DSM):Strategies used to manage
to input data associated with the actual installed measure into the v energy demand,including energy efficiency,load management,
algorithm(s). fuel substitution,and load building.
Deemed Savings Value:An estimate of energy or demand savings Dependent Variable:Term used in regression analysis or other
for a single unit of an installed energy efficiency measure that(1)has analyses seeking to explain the relationship among variables to
been developed from data sources and analytical methods that are quantify the variable that is being explained by the other
widely considered acceptable for the measure and purpose,and(2) (independent)variables.
is applicable to the situation being evaluated. Individual parameters
or calculation methods can also be deemed;also called stipulated Direct Emissions:Emissions from sources within an entity's organiza-
savings value. tional boundaries that are owned or controlled by the entity,includ-
ing stationary combustion emissions, mobile combustion emissions,
Degree Days:For any individual day,an indication of how far that process emissions,and fugitive emissions. Direct emissions are the
day's average temperature departed from a fixed temperature, source of avoided emissions for thermal energy efficiency measures
usually 18.3°C/65°F. Heating degree days,which measure heating (e.g.,avoided emissions from burning natural gas in a water heater).
energy demand,quantify how far the average temperature fell below
65°F.Similarly,cooling degree days,which measure cooling energy Direct Install Program:An energy efficiency program design strategy
demand,quantify how far the temperature averaged above 65°F. In involving the direct installation of measures in customer premises
both cases,smaller values represent less energy demand;however, by a contractor sponsored by the program.Such programs generally
values below 0 are set equal to 0,because energy demand cannot involve one-for-one replacement of existing equipment with more
be negative. Furthermore,because energy demand is cumulative, efficient equipment,and may include a customer rebate.
degree day totals for periods exceeding one day are simply the sum
of each individual day's degree days total.Degree days are used in Discount Rate:An interest rate applied to a stream of future costs
calculations of heating and cooling loads and in evaluation regression and/or monetized benefits to convert those values to a common
analyses to adjust for differences in heating and cooling requirements period,typically the current or near-term year,to measure and
between baseline and project scenarios. reflect the time value of money. It is used in benefit-cost analysis to
determine the economic merits of proceeding with the proposed
Demand:The time rate of energy flow. It is the requirement for project,and in cost-effectiveness analysis to compare the value
energy consumption of energy source(s)by an energy using system of projects.The discount rate for any analysis is either a nominal
at a given instant or averaged over any designated interval of time. discount rate or a real discount rate.A nominal discount rate is used
Demand usually refers to the amount of electric energy used by in analytic situations when the values are in then-current or nominal
a customer or piece of equipment at a specific time,expressed in dollars(reflecting anticipated inflation rates).A real discount rate
kilowatts(kW equals kWh/h)but can also refer to natural gas use at a is used when the future values are in constant dollars,and can be
point in time,usually as Btu/hr,kBtu/hr,therms/day,or cubic feet per approximated by subtracting expected inflation from a nominal
day(ccf/day). discount rate.
Demand Response(DR) The reduction of consumer energy use at Diversity:That characteristic of a variety of electric loads whereby
times of peak use in order to help system reliability, reflect market individual maximum demands of each load usually occur at
conditions and pricing,or support infrastructure optimization or different times.
deferral of additional infrastructure.Demand response programs may
include contractually obligated or voluntary curtailment,direct load Diversity Factor:The ratio of the sum of the demands of a group of
control,and pricing strategies. users to their coincident maximum demand during a specified period
of time(e.g.,summer or winter).
Demand Savings:The reduction in electric or gas demand from
the baseline to the demand associated with the higher-efficiency Education Programs:Programs primarily intended to educate
equipment or installation.This term is usually applied to billing customers about energy efficient technologies or behaviors or
demand to calculate cost savings or peak demand for equipment provide information about programs that offer energy efficiency
sizing purposes. or load-reduction information or services.These programs may
provide indirect energy or demand savings.
December 2012 www.seeaction.energy.gov A-5

Effective Useful Life(EUL):An estimate of the duration of savings systems,or operations that reduces the amount of energy that
from a measure. EUL is estimated through various means,including would otherwise have been used to deliver an equivalent or
median number of years that the energy efficiency measures improved level of end-use service.
installed under a program are still in place and operable.Also, EUL
is sometimes defined as the date at which 50%of Installed units are Energy Savings:Reduction in electricity use in kilowatt-hours or in
still in place and operational.See measure life. fossil fuel use in thermal unit(s).
Efficiency:The ratio of the useful energy delivered by a dynamic ENERGY STAR':A joint program of the U.S.Environmental Protection
system(e.g.,a machine,engine,or motor)to the energy supplied Agency and the U.S.Department of Energy designed to reduce energy
to it over the same period or cycle of operation.The ratio is usually use and the impact on the environment.The ENERGY STAR label is
determined under specific test conditions. awarded to products that meet applicable energy efficiency guide-
lines and to homes and commercial buildings that meet specified
Emission Factor:A representative value that relates the quantity energy efficiency standards.The program provides a range of energy
of a pollutant released to the atmosphere with an activity associ- management tools,primarily computer-based,for businesses.
ated with the release of that pollutant.These factors are usually
expressed as the weight of pollutant divided by a unit weight, Engineering Methods:The use of standard formulas or models
volume,distance,or duration of the activity emitting the pollutant based on those formulas,typically accepted by ASHRAE,as the basis
(e.g.,pounds of a pollutant per million Btu of heat input or pounds for calculating energy use.
of emissions per MWh of electricity produced).Such factors facilitate
estimation of emissions from various sources of air pollution.They Engineering Model: Engineering equations used to calculate energy
are based on available data of acceptable quality,and are generally use and savings.These models are usually based on a quantitative
assumed to be representative of averages for that pollutant. description of physical processes that transform delivered energy
into useful work,such as heat, lighting,or motor drive.In practice,
End Use:General categories of energy efficiency measures these models may be reduced to simple equations in spreadsheets
reflecting the type of services provided(e.g.,lighting,HVAC, that calculate energy use or savings as a function of measurable
motors,and refrigeration). attributes of customers,facilities,or equipment(e.g.,lighting use=
watts x hours of use).
End-Use Metering:The direct measuring of energy consumption
or demand by specific end-use equipment,typically as part of load Equipment Life:The number of years that a measure is installed and
research studies or to measure the impacts of demand-side operates until failure.
management programs.
Error:The deviation of measurements from the true value of the
Energy Conservation:Term used to reflect doing with less of a variable being observed;also called measurement error.
service in order to save energy.The term is often unintentionally
used instead of energy efficiency. Evaluated Savings:Savings estimates reported by an independent,
third-party evaluator after the subject energy efficiency activities
Energy Efficiency:The use of less energy to provide the same or an have been implemented and an impact evaluation has been
improved level of service to the energy consumer;or,the use of less completed;also called ex post,or more appropriately,ex post
energy to perform the same function. evaluated savings.
Energy Efficiency Activity:Any of a wide range of actions that are The designation of"independent"and"third party"is determined by
anticipated to result in the more efficient use of energy.Energy those entities involved in the use of the evaluations and may include
efficiency measures are a subset of energy efficiency activities,as evaluators retained,for example,by the administrator or
energy efficiency activities include non-technology specific actions a regulator.These values may rely on claimed savings for factors
such as education. such as installation rates and a technical reference manual for
values such as per-unit savings values and operating hours.These
Energy Efficiency Measure:At an end-use energy consumer facility, saving estimates may also include adjustments to claimed savings
an installed piece of equipment or system;a strategy intended to or projected savings for data errors,per-unit savings values,
affect consumer energy use behaviors;or modification of equipment,
A-6 www.seeaction.energy.gov December 2012

operating hours,installation rates,savings persistence rates,or which the participant's activity would have partially replicated the
other considerations. Evaluated savings can be indicated as first-year, program measure;or(3)deferred,in which the participant's activity
annual,and/or lifetime energy or demand savings values.They also would have partially or completely replicated the program measure,
can be indicated as gross savings and/or net savings values. but at a future time beyond the program's time frame.
Evaluation:The conduct of any of a wide range of assessment Fuel Switching: Using an alternative fuel(usually of lower carbon
studies and other activities aimed at determining the effects of a intensity)to produce required energy.
program and understanding or documenting program performance,
program or program-related markets and market operations, The change in energy consumption and/or
program-induced changes in energy efficiency markets,levels of demand that results from energy efficiency programs,codes and
demand or energy savings,or program cost-effectiveness. Market standards,and naturally occurring adoption,which have a long-
assessment,monitoring and evaluation,and measurement and lasting savings effect.Gross market savings generally do not include
verification are aspects of evaluation. temporary reductions in energy use from changes in weather,
income,energy prices,and other structural economic changes such
Evaluator:A person or entity that conducts evaluations. as in industry composition.
Ex Ante Savings:See projected savings. Gross Savings:The change in energy consumption and/or demand
that results directly from program-related actions taken by
Experimental Design:A method of estimating the impact of a participants in an energy efficiency program,regardless of why they
program or other event(e.g.,a medication,a procedure),in participated.
which outcomes between at least two randomly assigned groups
are compared. Heating Degree Days:The cumulative number of degrees in a
month or year by which the mean temperature falls below a fixed
Ex Post Savings:See claimed savings and evaluated savings. temperature,usually 18.3°C/65°F.See degree days.
Ex Post Evaluated Savings:See evaluated savings. Home Energy Rating System(HERS):An indexing system,associated
with ENERGY STAR®,used in residential new construction to rate the
External Validity:The condition in which an impact estimate that is pre-and post-construction of new homes to highlight and indicate
internally valid for a given program population and time frame can the degree of energy efficiency embedded in the construction.The
be generalized and applied to new situations(e.g.,new populations, HERS Index is a scoring system established by the Residential Energy
future years). Services Network(RESNET)in which a home built to the specifica-
FEMP M&V Guidelines:U.S. Department of Energy Federal Energy tions of the HERS Reference Home(based on the 2006 International
Management Program's 2008 M&V Guidelines:Measurement and Energy Conservation Code)scores a HERS Index of 100,while a net
Verification for Federal Energy Projects. zero energy home scores a HERS Index of 0.The lower a home's
HERS Index,the more energy efficient it is in comparison to the
Free Driver,Non-Participant:A program non-participant who has HERS Reference Home. Each 1-point decrease in the HERS Index
adopted particular energy efficiency measure(s)or practice(s)as a corresponds to a 1%reduction in energy consumption compared
result of the evaluated program.See spillover. to the HERS Reference Home.
Free Driver,Participant:A program participant who has adopted HVAC:Heating,ventilation,and air conditioning.
additional or incremental energy efficiency measure(s)or practice(s)
as a result of the evaluated program, but which were not directly Impact Evaluation:An evaluation of the program-specific,directly or
induced by the program.See spillover. indirectly induced changes(e.g.,changes in energy and/or demand
use)associated with an energy efficiency program.
Free Rider:A program participant who would have implemented the
program's measure(s)or practice(s)in the absence of the program. An entity selected and contracted with or qualified
Free riders can be(1)total,in which the participant's activity would by a program administrator to provide products and services to
have completely replicated the program measure;(2)partial,in consumers either directly or indirectly.
December 2012 www.seeaction.energy.gov A-7

Incentive:A financial strategy intended to encourage a change in number of measures installed by the number of measures incented
behavior related to energy use. Incentives can take various forms. by an energy efficiency program in a defined period of time.
Customer incentives are commonly used in energy efficiency pro-
grams as rebates for individual measures or as buy-downs in more Interactive Effects:The influence of one technology's application on
custom-oriented projects. Performance or shareholder incentives the energy required to operate another application.An example is
are monies that are established in a planning period to encourage the reduced heat in a facility as a result of replacing incandescent
program administrators to attain specified levels of savings during lights with CFLs,and the resulting need to increase space heating
the program year. from another source,usually oil-or gas-fired.With respect to IPMVP
Options A and B,interactive effects in energy use or demand occur-
Incremental Annual Savings:The difference between the amount ring beyond the measurement boundary of the M&V analysis.
of energy savings acquired or planned to be acquired as a result of
energy efficiency activities in one year,and the amount of energy Internal Validity:Refers to how well an evaluation was conducted
savings acquired or planned to be acquired as a result of the energy (e.g.,design,how variables were measured,what was/wasn't
efficiency activities in the prior year. measured)and how confidently one can conclude that the observed
effect(s)were produced solely by the independent variable and not
Incremental Cost:The difference between the cost of existing or extraneous ones.For impact evaluations,this is related to whether
baseline equipment or service and the cost of alternative energy- the savings impacts are valid for the specific program being evalu-
efficient equipment or service. ated,the given program participant population,and the given time
frame of the evaluation.This is often compared to external validity.
Independent Variables:The explanatory factors(e.g.,weather or
occupancy)in a regression model that are assumed to affect the International Performance Measurement and Verification Protocol
variable under study(e.g.,energy use). (IPMVP):A guidance document with a framework and definitions
describing the four M&V approaches;a product of the Efficiency
Indirect Emissions:Emissions that are a consequence of activities Valuation Organization(www.evo-world.org).
that take place within the organizational boundaries of an entity,but
occur at sources owned or controlled by another entity.For example, In its broadest terms,the concept that the effect of an
emissions of electricity used by a manufacturing entity that occur activity or outcome expected to occur and remain within a defined
at a power plant represent the manufacturer's indirect emissions. boundary flows outside the boundary,leading to unintended results.
Indirect emissions are typically the source of avoided emissions for In efficiency programs,an example of leakage is when a measure is
electric energy efficiency measures. incented by a program(with the associated costs and assumed
savings)but is installed outside of the program's jurisdiction.In
Indirect Energy(Demand)Savings(Indirect Program Energy the context of air regulation,such as a cap-and-trade program,an
Savings):The use of the words"indirect savings"or"indirect example of leakage is a shift of electricity generation from sources
program savings"refers to programs that are typically information, subject to the cap-and-trade program to higher-emitting sources
education,marketing,or outreach programs in which the program's not subject to the program.Sometimes used interchangeably
actions are expected to result in energy savings achieved through the with secondary effects,although leakage is a more"global"issue,
actions of the customers exposed to the program's efforts,without whereas secondary,interactive effects tend to be considered
direct enrollment in an program that has energy-savings goals. within the facility where a project takes place.
Inspections.Site visits to facilities treated under an energy efficiency Levelized Cost:The result of a computational approach used to
program that document the existence,characteristics,and operation compare the cost of different projects or technologies.The stream
of baseline or project equipment and systems,as well as factors of each project's net costs is discounted to a single year using a
that affect energy use.Inspections may or may not include review of discount rate(creating a net present value)and divided by the
commissioning or retro-commissioning documentation. project's expected lifetime output(megawatt-hours or therms).
Installation Rate:The percentage of measures that are incented by Lifetime Cost Per Kilowatt-hour or Therm:The cost associated
an energy efficiency program that are actually installed in a defined with a piece of equipment(supply-side or demand-side),energy
period of time.The installation rate is calculated by dividing the efficiency program,or total portfolio during its expected life in
A-8 www.seeaction.energy.gov December 2012

relation to(divided by)the electricity or gas that it produces or saves flow is shown by arrows from one to the others in the program logic.
over its lifetime.The annual costs are usually discounted back to a It can also be displayed as a table,with the linear relationship
single year using an appropriate discount rate. presented by the rows in the table.
Lifetime Demand Savings:The expected demand savings over the Main Meter:The meter that measures the energy used for the
lifetime of an installed measure(s),project(s),or program(s).It may whole facility.There is at least one meter for each energy source and
be calculated by multiplying the annual peak demand reduction possibly more than one per source for large facilities.Typically,utility
associated with a subject measure(s)by the expected useful lifetime meters are used,but dataloggers may also be used as long as they
of the measure(s). It may include consideration of technical degrada- isolate the load for the facility being studied.When more than one
tion and possibly the rebound effect.Savings can be gross or net. meter per energy source exists for a facility,the main meter may be
For electricity,it can be expressed in units of kilowatt-years. considered the accumulation of all the meters involved.
Lifetime Energy Savings The expected energy savings over the Marginal Cost:The sum that has to be paid for the next increment of
lifetime of an installed measure(s),project(s),or program(s). It may be product or service.The marginal cost of electricity is the price to be
calculated by multiplying the annual energy use reduction associated paid for kilowatt-hours above and beyond those supplied by
with a subject measure(s)by the expected useful lifetime of the sub- presently available generating capacity.
ject measure(s).It may include consideration of technical degradation
and possibly the rebound effect.Savings can be gross or net. The commercial activity(e.g.,manufacturing,distributing,
buying,and selling)associated with products and services that affect
Load Factor:A percentage indicating the difference between the energy use.
amount of electricity or natural gas a consumer used during a given
time span and the amount that would have been used if the use had Market Assessmeni An analysis that provides an assessment of how
stayed at the consumer's highest demand level during the whole and how well a specific market or market segment is functioning
time.The term also means the percentage of capacity of an energy with respect to the definition of well-functioning markets or with
facility,such as a power plant or gas pipeline,that is used in a given respect to other specific policy objectives.A market assessment
period of time. It is also the ratio of the average load to the peak generally includes a characterization or description of the specific
load during a specified time interval. market or market segments,including a description of the types
and number of buyers and sellers in the market,the key actors that
Load Management:Steps taken to reduce power demand at peak influence the market,the type and number of transactions that
load times or to shift some of it to off-peak times. Load management occur on an annual basis,and the extent to which market participants
may coincide with peak hours,peak days,or peak seasons. Load consider energy efficiency an important part of these transactions.
management may be pursued by persuading consumers to modify This analysis may also include an assessment of whether a market
behavior or by using equipment that regulates some electric has been sufficiently transformed to justify a reduction or elimina-
consumption.This may lead to complete elimination of electric use tion of specific program interventions.Market assessment can be
during the period of interest(load shedding)and/or to an increase in blended with strategic planning analysis to produce recommended
electric demand in the off-peak hours as a result of shifting electric program designs or budgets.One particular kind of market assess-
use to that period(load shifting). ment effort is a baseline study,or the characterization of a market
before the commencement of a specific intervention in the market
Load Shapes:Representations such as graphs,tables,and databases for the purpose of guiding the intervention and/or assessing its
that show the time-of-use pattern of customer or equipment energy effectiveness later.
use.These are typically shown over a 24-hour or whole-year(8,760
hours)period. Any characteristic of the market for an energy-
related product,service,or practice that helps to explain the gap
Logic Model:The graphical representation of a program theory between the actual level of investment in,or practice of,energy
showing the connection among activities,their outputs,and subse- efficiency and an increased level that would appear to be cost
quent short-term,intermediate,and long-term outcomes.Often,the beneficial to the consumer.
logic model is displayed with these elements in boxes,and the causal
December 2012 www.seeaction.energy.gov A-9

Market Effect:A change in the structure of a market or the behavior Market Sectors:General types of markets that a program may target
of participants in a market that is reflective of an increase(or or in which a service offering may be placed. Market sectors include
decrease)in the adoption of energy efficient products,services, categories such as agricultural,commercial,industrial,government,
or practices and is causally related to market interventions(e.g., and institutional.
programs). Examples of market effects include increased levels of
awareness of energy efficient technologies among customers and Market Segments:A part of a market sector that can be grouped
suppliers,increased availability of energy efficient technologies together as a result of a characteristic similar to the group.For
through retail channels,reduced prices for energy efficient models, example,within the residential sector are market segments such as
build out of energy efficient model lines,and—the end goal— renters,owners,multifamily,and single-family.
increased market share for energy efficient goods,services,and
design practices. Market Theory:A theoretical description of how a market operates
relative to a specific program or set of programs designed to influ-
Market Effect Evaluation:An evaluation of the change in the ence that market. Market theories typically include the identification
structure or functioning of a market,or the behavior of participants of key market actors,information flows,and product flows through
in a market,that results from one or more program efforts.Typically, the market, relative to a program designed to change the way the
the resultant market or behavior change leads to an increase in the market operates.Market theories are typically grounded upon the
adoption of energy efficient products,services,or practices. information provided from a market assessment but can also be
based on other information. Market theories often describe how
Market Event:The broader circumstances under which a customer a program intervention can take advantage of the structure and
considers adopting an energy efficiency product,service,or practice. function of a market to transform the market.Market theories can
Types of market events include,but are not necessarily limited to(1) also describe the key barriers and benefits associated with a market
new construction,or the construction of a new building or facility; and describe how a program can exploit the benefits and overcome
(2)renovation,or the updating of an existing building or facility;(3) the barriers.
remodeling,or a change in an existing building;(4)replacement,or
the replacement of equipment,either as a result of an emergency Market Transformation:A reduction in market barriers resulting
such as equipment failure or as part of a broader planned event; from a market intervention,as evidenced by a set of market effects
and(5)retrofit,or the early replacement of equipment or refitting that is likely to last after the intervention has been withdrawn,
of a building or facility while equipment is still functioning,often as reduced,or changed.
a result of an intervention into energy efficiency markets.
Measure: [verb] Use of an instrument to assess a physical quantity
Market Participants The individuals and organizations participating or use of a computer simulation to estimate a physical quantity.
in transactions with one another within an energy efficiency market
or markets,including customers and market actors. Measure:[noun]See energy efficiency measure.
Market Penetration Rate:A measure of the diffusion of a technol- Measure Life:The length of time that a measure is expected to
ogy,product,or practice in a defined market,as represented by the be functional;sometimes referred to as expected useful life.
percentage of annual sales for a product or practice,the percentage Measure life is a function of equipment life and measure persistence.
of the existing installed stock for a product or category of products, Equipment life is the number of years that a measure is installed and
or the percentage of existing installed stock that uses a practice. will operate until failure.Measure persistence takes into account
business turnover,early retirement of installed equipment,and
Market Saturation:A percentage indicating the proportion of a other reasons measures might be removed or discontinued.
specified end-user market that contains a particular product.An
example would be the percentage of all households in a given Measurement and Verification(M&V):A subset of program impact
geographical area that have a certain appliance.Studies conducted evaluation that is associated with the documentation of energy
to obtain this information within the residential sector are referred savings at individual sites or projects using one or more methods
to as residential appliance saturation studies(RA55). that can involve measurements,engineering calculations,statistical
analyses,and/or computer simulation modeling. M&V approaches
are defined in the IPMVR
A-10 www.seeaction.energy.gov December 2012

Measurement Boundary:The boundary of the analysis for determin- Naturally Occurring Efficiency:The effects of energy-related deci-
ing direct energy and/or demand savings. sions,by both program participants and non-participants,that would
have been made in the absence of the program;alternatively,the
Measure Penetration:The fraction of annual market sales captured expected average efficiency of one or more measures or systems in
by a more efficient measure or system at a given point in time;also the absence of all publicly funded energy efficiency programs. It can
called market penetration. For example,the market penetration of be part of a baseline determination.
CFLs was 20%in the Florida market in 2005.
Net Present Value(NPV):The value of a stream of cash flows
Measure Persistence:The duration of an energy-consuming converted to a single sum in a specific year,usually the first year of
measure,taking into account business turnover,early retirement the analysis. It can also be thought of as the equivalent worth of all
of installed equipment,technical degradation factors,and other cash flows relative to a base point called the present.
reasons measures might be removed or discontinued.
The change in energy consumption and/or demand that
Measure Retention Study:An assessment of(1)the length of time is attributable to a particular energy efficiency program.This change
the measure(s)installed during the program year are maintained in in energy use and/or demand may include,implicitly or explicitly,
operating condition,and(2)the extent to which there has been a consideration of factors such as free ridership,participant and non-
significant reduction in the effectiveness of the measure(s). participant spillover,and induced market effects.These factors may
be considered in how a baseline is defined(e.g.,common practice)
Measure Saturation:The fraction of a total market(e.g.,consumers, and/or in adjustments to gross savings values.
buildings)captured by a specific efficiency activity at a given point in
time;also called market saturation. For example,in 2005,20%of the Net-to-Gross(NTG)Ratio:A factor representing net program
commercial buildings that are more than 100,000 square feet in savings divided by gross program savings that is applied to gross
Portland had high-efficiency chillers that exceeded minimum program impacts to convert them into net program load impacts.
efficiency standards. The factor itself may be made up of a variety of factors that create
differences between gross and net savings,commonly including free
Metered Data:Data collected over time through a meter for a riders and spillover.Can be applied separately to either energy or
specific energy using,end-use system(e.g.,lighting and HVAC), demand savings.
or location(e.g.,floors of a building or a whole premise). Metered
data may be collected over a variety of time intervals. Metered New Construction: Residential and nonresidential buildings that
data usually refers to electricity or gas data. have been newly built or have added major additions.
Metering:The collection of energy-consumption data over time Nominal:For dollars,"nominal"means the figure representing the
through the use of meters.These meters may collect information actual number of dollars exchanged in each year,without accounting
with respect to an end use,a circuit,a piece of equipment,or a for the effect of inflation on the value or purchasing power.For inter-
whole building(or facility).Short-term metering generally refers to est or discount rates,"nominal"means that the rate includes the
data collection for no more than a few weeks.End-use metering rate of inflation(the real rate plus the inflation rate approximately
refers specifically to separate data collection for one or more end equals the nominal rate).
uses in a facility,such as lighting,air conditioning,or refrigeration.
Spot metering is an instantaneous measurement(rather than over Non-Energy Effects or Non-Energy Benefits(NEB):The identifi-
time)to determine an energy consumption rate. able non-energy impacts associated with program implementation
or participation;also referred to as non-energy impacts(NEI)or
The collection of relevant measurement data over time co-benefits.Examples of NEBs include avoided emissions and other
at a facility,including but not limited to energy consumption or environmental benefits,productivity improvements,jobs created,
emissions data(e.g.,energy and water consumption,temperature, reduced program administrator debt and disconnects,and higher
humidity,volume of emissions,hours of operation)for the purpose comfort and convenience level of the participant.The value is most
of savings analysis or to evaluate equipment or system performance. often positive,but may also be negative(e.g.,the cost of additional
maintenance associated with a sophisticated,energy-efficient
control system).
December 2012 www.seeaction.energy.gov A-11

Non-Participant:Any consumer who was eligible but did not programs),or(2)the set of all programs conducted by one
participate in the subject efficiency program,in a given program year. organization,such as a utility(and which could include programs that
cover multiple markets,technologies,etc.).
Normalized Annual Consumption(NAC)Analysis:A regression-
based method that analyzes monthly energy consumption data and Potential,Achievable:The amount of energy or demand savings
adjusts the consumption data to eliminate annual or other periodic within a defined geographical area or population that can be
fluctuations in an influencing factor(e.g.,weather on heating and achieved in response to specific energy efficiency program designs,
cooling needs)based on a historical normal or average pattern of the delivery approaches,program funding,and measure incentive levels;
influencing factor. sometimes referred to as market potential,program potential,or
realistic potential.
Offset: Program mechanism that allows an entity to neutralize a
portion or all of its regulated,capped emissions contribution by Potential,Economic:Refers to the subset of the technical
orchestrating or funding projects that are not subject to regulation potential that is economically cost-effective as compared to
(i.e.,an emissions cap). conventional options.
Panel Data Model:An estimation analysis model that contains many Potential Studies:Studies conducted to assess market baselines and
data points over time rather than averaged,summed,or otherwise future savings that may be expected for different technologies and
aggregated data. customer markets over a specified time horizon.
Participant:A consumer that received a service offered through Potential,Technical:An estimate of energy savings based on the
the subject efficiency program,in a given program year;also called assumption that all existing equipment or measures will be replaced
with the most efficient equipment or measure that is both available
program participant.The term"service"is used in this definition
to suggest that the service can be a wide variety of inducements, and technically feasible over a defined time horizon,without regard
including financial rebates,technical assistance,product installations, to cost or market acceptance.
training,energy efficiency information or other services,items,or Precision:The indication of the closeness of agreement among
conditions. Each evaluation plan should define"participant"as it repeated measurements of the same physical quantity. Precision
applies to the specific evaluation. is a measure of how statistically confident evaluators can be that
the estimated impact of a program is close to the true impact of a
Participant Cost Test(PCT):A cost-effectiveness test that measures program.An estimate with a smaller confidence interval is said to
the economic impact to the participant of adopting an energy be more precise. It is also used to represent the degree to which
efficiency measure. an estimated result in social science(e.g.,energy savings)would be
Peak Demand:The maximum level of hourly demand during a replicated with repeated studies.
specified period.The peak periods most commonly identified are Prescriptive Program:An energy efficiency program focused
annual and seasonal(summer and winter). on measures that are one-for-one replacements of the existing
equipment and for which fixed customer incentives can be
Peak Load:The highest electrical demand within a particular period developed based on the anticipated similar savings that will accrue
of time.Daily electric peaks on weekdays typically occur in late from their installation.
afternoon and early evening.Annual peaks typically occur on hot
summer days. Primary Effects:Effects that the project or program are intended
to achieve.For efficiency programs,this is primarily a reduction in
Persistence:See savings persistence rate and measure persistence. energy use(and/or demand)per-unit of output.
Persistence Study:A study to assess changes in program impacts Process Evaluation:A systematic assessment of an energy efficiency
over time.See savings persistence rate and measure persistence. program for the purposes of documenting program operations at
Portfolio:Either(1)a collection of similar programs addressing the the time of the examination,and identifying and recommending
same market(e.g.,a portfolio of residential programs),technology improvements to increase the program's efficiency or effectiveness
(e.g.,motor efficiency programs),or mechanisms(e.g.,loan for acquiring energy resources while maintaining high levels of
participant satisfaction.
A-12 www.seeaction.energy.gov December 2012

Program:An activity,strategy,or course of action undertaken by an willingness to sign up for the program). Randomization creates a
implementer or administrator.Each program is defined by a unique control group that is statistically identical to the treatment group,
combination of the program strategy, market segment,marketing in both observable and unobservable characteristics,such that any
approach,and energy efficiency measure(s)included. Programs difference in outcomes between the two groups can be attributed to
consist of a group of projects with similar characteristics and the treatment with a high degree of confidence.
installed in similar applications.
Randomized Controlled Trial(RCT):A type of experimental program
Program Administrator Cost Test(PACT, See utility/program evaluation design in which energy consumers in a given population
administrator cost test. are randomly assigned into two groups:a treatment group and a
control group.The outcomes for these two groups are compared,
Program Incentive:An incentive,generally monetary,that is offered resulting in program energy savings estimates.
to a customer through an energy efficiency program to encourage
the customer to participate in the program.The incentive is intended Ratepayer Impact Measure(RIM)Test:A cost-effectiveness test that
to overcome one or more barriers that keep the customer from measures the impact on utility operating margin and whether rates
taking the energy efficiency activity on his own. would have to increase to maintain the current levels of margin if
a customer installed energy efficient measures.The test measures
Program Theory:A presentation of the goals of a program, what happens to customer bills or rates due to changes in utility
incorporated with a detailed presentation of the activities that the revenues and operating costs caused by the program.
program will use to accomplish those goals and the identification of
the causal relationships between the activities and the program's For dollars,"real"means that the dollars are expressed in
effects. Program theory is often the basis for the logic model. a specific base year in order to provide a consistent means of
comparison after accounting for inflation.For interest and discount
Program Year(PY):The calendar year approved for program rates,"real"means the inflation rate is not included(the nominal
implementation. Note that program years can be shorter than rate minus the inflation rate approximately equals the real rate).
12 months if programs are initiated mid-year.
Realization Rate:Used in several contexts for comparing one savings
Project:An activity or course of action involving one or multiple estimate with another.The primary and most meaningful application
energy efficiency measures at a single facility or site. is the ratio of evaluated gross savings to claimed gross savings
(versus comparing net and gross savings estimates,which are best
Projected Savings:Values reported by an implementer or defined with a net-to-gross ratio). Basis for the ratio not being 1.0
administrator before the subject energy efficiency activities are can include several considerations,such as(1)adjustments for data
implemented;also called planning estimates or ex ante estimates. errors,(2)differences in implemented measure counts as a result of
Projected savings are typically estimates of savings prepared for verification activities,and/or(3)other differences revealed through
program and/or portfolio design or planning purposes.These values the evaluation process,such as changes in baseline assumptions.
are typically based on pre-program or portfolio estimates of factors Rebate:See incentive.
such as per-unit savings values,operating hours,installation rates,
and savings persistence rates.These values may use results of prior Rebate Progran An energy efficiency program in which the program
evaluations and/or values in a technical reference manual.They can administrator offers a financial incentive for the installation of
be indicated as first-year,annual,and/or lifetime energy or demand energy-efficient equipment.
savings values.They can be indicated as gross savings and/or net
savings values. Rebound Effect:A change in energy-using behavior that yields an
increased level of service accompanied by an increase in energy
Random Assignment:A condition where each household or entity use that occurs as a result of taking an energy efficiency action;
in the study population is randomly assigned to either the control also called take back.The result of this effect is that the absolute
group or the treatment group based on a random probability,as savings associated with the direct energy efficiency action is
opposed to being assigned to one group or the other based on impacted(usually reduced)by the resulting behavioral change.
some characteristic of the household(e.g.,location,energy use,or The rebound effect can be considered as a factor in determining
savings persistence rate.
December 2012 www.seeaction.energy.gov A-13

Regression Analysis:Analysis of the relationship between a to isolate the energy flows for the system(s)under consideration.
dependent variable(response variable)to specified independent IPMVP Option A involves"Key Parameter Measurement"and IPMVP
variables(explanatory variables).The mathematical model of their Option B involves'All Parameter Measurement."
relationship
is the regression equation. Retrofit Program:An energy efficiency program that provides
incentives,information,and technical support to encourage
Regression Model:A mathematical model based on statistical customers to replace existing and operating equipment or systems
analysis where the dependent variable is quantified based on its with more energy-efficient equipment or systems that provides the
relationship to the independent variables that are said to determine same function,or to add energy efficient equipment or systems to an
its value. In so doing,the relationship between the variables is existing facility(e.g.,addition of thermal insulation).
estimated statistically from the data used.
Rigor:The level of expected confidence and precision.The higher
Reliability:The quality of a measurement process that would the level of rigor,the more confident one is that the results of the
produce similar results on(1)repeated observations of the same evaluation are reliable.
condition or event,or(2)multiple observations of the same
condition or event by different observers.Reliability refers to the Sample:In program evaluation,a portion of the population selected
likelihood that the observations can be replicated. to represent the whole. Differing evaluation approaches rely on
simple or stratified(based on some characteristic of the population)
Reporting Period:The time following implementation of an energy samples.
efficiency activity during which savings are to be determined.
Sample Design:The approach used to select the sample units.
Representative Sample:A sample that has approximately the same
distribution of characteristics as the population from which it was Sampling Error:An error that arises because the data are collected
drawn.Such samples may be randomly selected or not. Random from a part,rather than the whole of the population. It is usually
samples selected from the same population as participants are more measurable from the sample data in the case of probability sampling.
representative of the program participants.
Savings Persistence Rat Percentage of first-year energy or demand
Resource Acquisition Program:A program designed to achieve savings expected to persist over the life of the installed energy
directly energy and/or demand savings;also called retrofit program. efficiency equipment.It is developed by conducting surveys of
Such a program generally involves encouraging customers to replace installed equipment several years after installation to determine
existing equipment with more-efficient equipment. presence and operational capability of the equipment.
Retro-commissioning:Often abbreviated as"RCx,"a systematic Seasonal Energy Efficiency Ratio(SEER)-The total cooling output of
method for investigating how and why an existing building's systems a central air-conditioning unit(in Btu)during its normal use period
are operated and maintained,and for identifying ways to improve for cooling divided by the total electrical energy input(in watt-hours)
overall building performance. Retro-commissioning improves during the same period,as determined using specified federal
a building's operations and maintenance(0&M)procedures to test procedures.
enhance overall building performance.
Secondary Effects: Unintended impacts of the project or program
Energy efficiency activities undertaken in existing residential such as rebound effect(e.g.,increasing energy use as it becomes
or nonresidential buildings,where existing inefficient equipment or more efficient and less costly to use),activity shifting(e.g.,when
systems are replaced by energy-efficient equipment or systems or generation resources move to another location),and market leakage
where efficient equipment or systems are added to an existing (e.g.,emission changes due to changes in supply or demand of
facility(e.g.,addition of thermal insulation). commercial markets).Secondary effects can be positive or negative.
Retrofit Isolation:The savings measurement approach defined Snap Back:See rebound effect.
in IPMVP Options A and B,as well as ASHRAE Guideline 14,that
determines energy or demand savings through the use of meters Societal Cost Test(SCT):A cost-effectiveness test that measures the
net economic benefit to the utility service territory,state,or region,
A-14 www.seeaction.energy.gov December 2012

as measured by the total resource cost test, plus indirect benefits Structured Interview:An interview in which the questions to be
such as environmental benefits. asked,their sequence,and the detailed information to be gathered
are all predetermined.These are used where maximum consistency
Spillover(Participant and Non-Participant): Reductions in energy across interviews and interviewees is needed.
consumption and/or demand caused by the presence of an energy
efficiency program,beyond the program-related gross savings of Take Back Effect:See rebound effect.
the participants and without direct financial or technical assistance
from the program.There can be participant and/or non-participant Technical Degradation Factor:A multiplier used to account for time-
spillover.Participant spillover is the additional energy savings and use-related change in the energy savings of a high-efficiency
that occur as a result of the program's influence when a program measure or practice relative to a standard-efficiency measure or
participant independently installs incremental energy efficiency practice due to technical operational characteristics of the measures,
measures or applies energy-saving practices after having participated including operating conditions and product design.
in the energy efficiency program.Non-participant spillover refers to
energy savings that occur when a program non-participant installs Technical Reference Manual(TRM):A resource document that
energy efficiency measures or applies energy savings practices as a includes information used in program planning and reporting
result of a program's influence. of energy efficiency programs. It can include savings values for
measures,engineering algorithms to calculate savings,impact factors
Spillover Rate: Estimate of energy savings attributable to spillover to be applied to calculated savings(e.g.,net-to-gross ratio values),
effects expressed as a percent of savings installed by participants source documentation,specified assumptions,and other relevant
through an energy efficiency program. material to support the calculation of measure and program
savings—and the application of such values and algorithms in
Statistically Adjusted Engineering(SAE)Models:A category of appropriate applications.
statistical analysis models that incorporate the engineering estimate of
savings as a dependent variable.The regression coefficient in these Total Resource Cost(TRC)Test:A cost-effectiveness test that
models is the percentage of the engineering estimate of savings measures the net direct economic impact to the utility service
observed in changes in energy use. For example,if the coefficient on territory,state,or region.
the SAE term is 0.8,this means that the customers are on average
realizing 80%of the savings from their engineering estimates. Tracking Estimate:See claimed savings.
Stipulated Values:See deemed savings. Treatment Group:The group of consumers that receive the subject
program's intervention or"treatment"(i.e.,the program participants).
Stratified Random Sampling:A sampling method where the
population is divided into X units of subpopulations,called Uncertainty:The range or interval of doubt surrounding a measured
strata,that are non-overlapping and together make up the entire or calculated value within which the true value is expected to fall
population.A simple random sample is taken of each strata to create within some degree of confidence.
a sample based upon stratified random sampling.
Upstream Program:A program that provides information and/or
Stratified Ratio Estimation:A sampling method that combines financial assistance to entities in the delivery chain of high-efficiency
a stratified sample design with a ratio estimator to reduce the products at the retail,wholesale,or manufacturing level.Such a
coefficient of variation by using the correlation of a known measure program is intended to yield lower retail prices for the products.
for the unit(e.g.,expected energy savings)to stratify the population
Utility/Program Administrator Cost Test:Acost-effectiveness test
and allocate samples from strata for optimal sampling.Stratified
that measures the change in the amount the utility must collect from
ratio estimation can reduce the number of sites,observations,and
the customers every year to meet an earnings target(e.g.,a change
thus evaluation costs required to achieve a given level of precision by
in revenue requirement);also called program administrator cost test
using the correlation of a known measure for the unit(e.g.,expected
(PACT)and also known as the utility cost test. In several states,this
energy savings)to stratify the population and allocate sample
test is referred to as the program administrator cost test. In those
requirements for each strata.
cases,the definition of the"utility"is expanded to program
administrators(utility or third-party).
December 2012 www.seeaction.energy.gov A-15

Verification:An independent assessment that the program has been
implemented per the program design.For example,the objectives
of measure installation verification are to confirm(1)the installation
rate,(2)that the installation meets reasonable quality standards,and
(3)that the measures are operating correctly and have the potential
to generate the predicted savings.Verification activities are generally
conducted during on-site surveys of a sample of projects.Project site
inspections,participant phone and mail surveys,and/or implementer
and consumer documentation review are typical activities associated
with verification.Verification may include one-time or multiple
activities over the estimated life of the measures. It may include
review of commissioning or retro-commissioning documentation.
Verification is a subset of evaluation and,as such,can also include
review and confirmation of evaluation methods used,samples
drawn,and calculations used to estimate program savings.
Whole-Building Calibrated Simulation Approach:A savings
measurement approach(defined in IPMVP Option D and ASHRAE
Guideline 14)that involves the use of an approved computer
simulation program to develop a physical model of the building in
order to determine energy and demand savings.The simulation
program is used to model the energy used by the facility before and
after the retrofit.The pre-or post-retrofit models are calibrated with
measured energy use and demand data as well as weather data.
Whole-Building Metered Approach:A savings measurement
approach(defined in the IPMVP Option C and ASHRAE Guideline
14)that determines energy and demand savings through the use
of whole-facility energy(end-use)data,which may be measured by
utility meters or data loggers.This approach may involve the use of
monthly utility billing data or data gathered more frequently from a
main meter.
Workforce Education and Training Program Programs primarily
intended for building maintenance engineers, HVAC contractors,
engineers,architects,maintenance personnel,and others.These
programs provide information about energy efficiency concepts,
recommended energy-efficient technologies or behaviors,and/or
programs that offer energy efficiency or load-reduction information,
products or services.These programs may provide indirect energy or
demand savings.
A-16 www.seeaction.energy.gov December 2012

Glossary References
California Commissioning Collaborative. (2006). California Northeast Energy Efficiency Partnerships. (July 2011).EM&V Forum
Commissioning Guide:New Buildings and California GLOSSARY OF TERMS Version 2.1.PAH Associates.A project of the
Commissioning Guide:Existing Buildings. www.cacx.org/ Regional Evaluation,Measurement and Verification Forum.
resources/commissioning-guides.html. http://neep.orgluploadslEMV°o20ForumlEMVYa20ProductsI
EMV Glossary-Version_2.1.pdf.
California Public Utilities Commission.(April 2006).California Energy
Efficiency Evaluation Protocols:Technical, Methodological,and State and Local Energy Efficiency Action Network. (May 2012).
Reporting Requirements for Evaluation Professionals. Prepared Evaluation,Measurement,and Verification(EM&V)of Residential
by The TecMarket Works Team. www.calmac.org/events/ Behavior-Based Energy Efficiency Programs:Issues and
EvaluatorsProtocols_Final AdoptedviaRuling_06-19-2006.pdf. Recommendations.Prepared by Todd,A.;Stuart,E.;Schiller,S.;
Goldman, C.;Lawrence Berkeley National Laboratory.
Efficiency Valuation Organization. (September2010).International wwwl.eere.energy.gov/seeaction/pdfs/emv_behaviorbased
Performance Measurement and Verification Protocol.EVO eeprograms.pdf.
10000-1:2010. Concepts and Options for Determining Energy
and Water Savings Volume 1. www.evo-world.org. World Resources Institute and World Business Council for
Sustainable Development. (November 2005). The GHG Protocol
Eto,J.;Prahl,R.;Schegel,J. (1996).A Scoping Study on Energy- for Project Accounting. www.ghgprotocol.org/files/ghgp/
Efficiency Market Transformation by California Utility DSM ghg_project protocol.pdf.
Programs. Lawrence Berkeley National Laboratory.
http://eetd.lbl.gov/ea/ems/reports/39058.pdf.
National Action Plan for Energy Efficiency. (2007). Guide for Conduct-
ing Energy Efficiency Potential Studies. Optimal Energy,Inc.
www.epa.gov/cleanenergy/documents/suca/potential guide.pdf.
National Action Plan for Energy Efficiency. (2007).Model Energy
Efficiency Program Impact Evaluation Guide.Steven R.Schiller,
Schiller Consulting, Inc. www.epa.gov/eeactionplan.
National Action Plan for Energy Efficiency. (2008). Understanding
Cost-Effectiveness of Energy Efficiency Programs:Best Practices,
Technical Methods,and Emerging Issues for Policy-Makers.
Prepared by Energy and Environmental Economics,Inc. (E3)and
Regulatory Assistance Project(RAP). www.epa.gov/cleonenergy/
documents/suca/cost-effectiveness.pdf
National Action Plan for Energy Efficiency. (2009).Energy Efficiency
as a Low-Cost Resource for Achieving Carbon Emissions
Reductions.ICF International,Inc. www.epa.gov/cleanenergy/
documents/suca/ee_and carbon.pdf.
December 2012 www.seeaction.energy.gov A-17

Appendix B: Other Evaluation Categories
and Approaches
This appendix provides a brief introduction to process and market 13,1.1 Process Evaluations
effects evaluations,cost-effectiveness analysis,and impact evalua- The goal of process evaluations is to produce better and more cost-
tions using"top-down"approaches.The material in this appendix is effective programs. Process evaluations meet this goal by assessing
intended to supplement the other sections of the guide,which are the processes a program undergoes during implementation,docu-
focused on"bottom-up"impact evaluations. menting program goals and objectives from a variety of perspectives,
B.1 PROCESS, MARKET EFFECTS, AND and describing program strengths and weaknesses so that success
COST EFFECTIVENESS EVALUATIONS is highlighted and improvements can be made in a timely manner.
Thus, process evaluations examine the efficiency and effectiveness
The following subsections introduce three non-impact types of evalu- of program implementation procedures and systems.Typical process
ations:process,market,and cost-effectiveness. However,because evaluation results involve recommendations for changing a pro-
cost-effectiveness analyses rely on the documentation of program gram's structure,implementation approaches,and goals.
impacts,these analyses are often considered a component of impact
evaluations,and program cost-effectiveness indicators are thus often These evaluations usually consist of asking questions of those
included in impact evaluation reports.Table 13.1 compares these involved in the program,analyzing their answers,and comparing
three evaluation types,plus impact evaluations. results to established best practices.Whereas it is typically required
that an independent third-party evaluator is involved in conducting
TABLE BA: Program Evaluation Types
EVALUATION TYPE DESCRIPTION EXAMPLES USES
Impact Evaluation Quantifies direct and indirect changes Determines the amount of energy and
associated with the subject program(s) demand saved
Process Evaluation Indicates how the procedures associated Identifies how program designs and
with program design and implementation processes can be improved
are performing from both the administra-
tor's and the participants' perspectives
Market Effects Evaluation Analyzes how the overall supply chain Characterizes changes that have occurred
and market for energy efficiency products in efficiency markets and whether they
have been affected by the program are attributable to and sustainable with
or without the program
Cost-Effectiveness Evaluation Quantifies the costs of program Determines whether an energy efficiency
implementation and compares them program is a cost-effective investment
with program benefits compared with other programs and
energy supply resources
B-1 www.seeaction.energy.gov December 2012

impact evaluations,for process evaluations,jurisdictions might variety of outputs that lead to key outcomes(see sidebar in Chapter
recommend(but not require)them to be conducted by independent 7 on Theory-Based Evaluation:A Guiding Principle for MT Evaluation).
third-party evaluators; however,the use of third-party process evalu- Logic models can be linked to performance indicators that provide
ators is a best practice. Use of a trusted party for process evaluation ongoing feedback to program managers.The models usually flow top
is important for successful process evaluation so that the evaluator to bottom and are often organized according to five basic categories:
can gather the necessary data and provide feedback in a manner • Program inputs:financial,staffing,and infrastructure resources
that is productive(e.g.,not considered threatening by the recipient that support the activity
of the feedback).
• Program activities:overarching activities that describe what
Process evaluations are particularly valuable in the following situations: the program is doing(e.g.,marketing and rebate processing)
• Benefits are higher/lower than expected and/or are being • Outputs:metrics resulting from the activities,and that tend to
achieved more quickly/slowly than expected. be measurable"bean counting" results(e.g., provide outreach
• There is limited program participation or stakeholders are slow events at five community fairs)
to begin participating. • Short-to intermediate-term outcomes:expected outcomes
• The program is a greater success than anticipated. resulting from program activities,with goals attached to those
outcomes when possible.(e.g.,target energy savings and
• The program has a slow start-up. recruitment into the program)
• Participants are reporting problems. • Long-term outcomes and goals:ideal,sustainable outcomes
• The program appears not to be cost effective. resulting from program activities(e.g.,"all eligible customers
• The program is built around a new concept that could be participate in the program"and"increase customer awareness
replicable for other populations,technologies,etc. of program offerings").
As part of a process evaluation,a logic model may be developed for the These logic model categories indicate the intended and expected
program(or possibly a set of logic models for a complete portfolio of results of activities.Expected short-, medium-,and long-term out-
programs).A program's theory and logic model serve as a roadmap to comes tend to define program goals at a high level and also specify
guide the systematic approach of a process evaluation.A program logic market effects(i.e.,expected program outcomes). In this manner,
model is a visual representation of the program's theory that illustrates process evaluation is part of a continuum linking impact and market
a set of interrelated program activities that combine to produce a effects evaluations.
TABLE B.2: Elements of Typical Process Evaluations
Program Design Program Implementation
• The program mission • Quality control
• Assessment of program logic • Operational practice—how the program is implemented
• Use of new practices or best practices • Program targeting,marketing,and outreach efforts
• Program timing
Program Administration Participant Response
• Program oversight • Participant interaction and satisfaction
• Program staffing • Market and government allies interaction and satisfaction
• Management and staff training
• Program information and reporting
December 2012 www.seeaction.energy.gov B-2

The primary mechanism of process evaluations is data collection programs.In this guide,the definition of market transformation is:
(e.g.,surveys,questionnaires,and interviews)from administrators, a reduction in market barriers resulting from a market intervention,
designers,participants(e.g.,facility operators,business owners, as evidenced by a set of market effects,that is likely to last after the
renters,or homeowners),implementation staff(including contrac- intervention has been withdrawn, reduced,or changed.
tors,subcontractors,and field staff),trade allies(e.g.,mechanical
contractors,architects,and engineers)and key policymakers.Other Market effects evaluations often involve a significant undertaking,
elements of a process evaluation can include workflow and produc- because they require collection and analysis of data from a wide
tivity measurements;reviews,assessments,and testing of records, range of market actors,as well as analysis of those data against a
databases,program-related materials,and tools;and collection and background developed out of secondary sources.Market effects are
analysis of relevant data from third-party sources(e.g.,equipment sometimes called the ultimate test of a program's success,answering
vendors or retailers).Process evaluations can be operated continu- the question:"Will energy efficiency(best)practices continue in the
ously,perhaps as part of a continuous improvement effort,or at marketplace,even after the current program ends?"The difference
intervals(e.g.,as a new program is being implemented,whenever between a market change and a market effect is attribution:the
there are major changes in a program,in response to issues noted in ability to trace back a change in the market to a specific program or
first set of bullets above,and/or just every two to three years). group of programs.The following is a definition of market effects
from a well-referenced 1996 study:los
Table B.2 lists examples of program elements typically assessed
during a process evaluation. Market effect:a change in the structure of a market or the
behavior of market participants that is reflective of an increase in
B.1.2 Market Effects Evaluations the adoption of energy-efficient products,services,or practices
The goal of market effects evaluations is to characterize and quantify and is causally related to market intervention(s)(e.g.,programs).
the effects of a program on supplier promotion and customer Examples of market effects include increased levels of awareness
adoption of the targeted energy efficiency measures,regardless of of energy-efficient technologies among customers and suppli-
whether those suppliers and customers participated in the program. ers,increased availability of efficient technologies through retail
Effects that cannot be captured by program records are particularly channels,reduced prices for efficient models,build-out of efficient
important for certain kinds of initiatives,including"upstream" model lines,and—the end goal—increased market share for
promotions of mass-market goods,such as light bulbs and consumer efficient goods,services,and design practices.
electronics as well as training programs aimed at inducing engineers
and contractors to adopt energy efficiency design and specification POTENTIAL STUDIES
practices.Studies have shown that even straightforward equipment
rebate programs may have effects"outside the program"by Another form of market study(although not formally an
exposing contractors and large customers to the benefits of efficient "evaluation")is called a potential study.Potential studies
technologies.This in turn leads to increased specification of efficient are conducted before a program is implemented in order
technologies on projects that do not receive program support. In to assess market baselines and future savings potentials for
some cases, market effects evaluation results can be combined with different efficiency technologies,strategies,or approaches
impact evaluation findings to estimate program-induced energy in different customer markets.These studies can also assess
savings that were not tracked by the program itself. customer needs and barriers to adoption of energy efficiency,
Other market studies include potential studies(see sidebar)and as well as how best to address these barriers through pro-
market baseline studies. Potential studies investigate how much gram design. Potential studies indicate what can be expected
saving may be available through various measures and baseline in terms of savings from a program. Potential is often defined
studies look at indicators of market development before the in terms of technical potential(what is technically feasible
program intervention. given commercially available products and services),and
economic potential(which is the level of savings that can
Market effects studies are usually associated with programs that be achieved assuming a certain level of participant and/or
have a specific market transformation focus.There are many defini- societal cost effectiveness is required).Findings also help
tions of market transformation,although it is often considered the managers identify the program's key markets and clients and
ultimate goal of publicly and consumer-funded energy efficiency how to best serve the intended customers.
B-3 www.seeaction.energy.gov December 2012

Examples of the questions that a market effects evaluation might without several years of information;even then,they will usually
answer are as follows: not have the same level of accuracy as impact evaluations of direct
• Are the entities that undertook energy efficiency projects resource acquisition savings.To credit measure adoption and
undertaking additional projects or incorporating additional associated savings to a program,it must be shown that the increased
technologies in their facilities that were not directly induced energy efficiency adoption,the longer-term market effects,and the
by the program?This might indicate that facility operators have participant effects have all occurred essentially in the manner and
become convinced of the value of,for example, high-efficiency in the order specified by the program theory.And this,for most
motors,and are installing them on their own. programs,takes a number of years to reach this point.
• Are entities that did not undertake projects now adopting In 2009,a comprehensive white paper study on market transformation
concepts and technologies that were encouraged by the and market effects was prepared.106 Table 13.3,from a presentation
program?This might indicate that the program convinced by the principle author of this white paper,indicates approaches for
other facility operators of the advantages of the energy assessing market effects,including attribution.
efficiency concepts.
• Are manufacturers,distributors,retailers,vendors,and others As can be deduced from the above discussion and Table 13.3,the
involved in the supply chain of energy efficiency products market effects evaluation can easily overlap with the spillover analyses
(and services)changing their product offerings—for example, conducted as part of an impact evaluation. In fact,many of the tech-
how are they marketing them,pricing them,stocking them? niques used to quantify market effects can be applied in estimating
The answers can indicate how the supply chain is adapting to spillover savings.
changes in supply of and demand for efficiency products. 107,108
B.1.3 Cost-Effectiveness Analyses
Structuring a market effects evaluation entails consideration of sev- Cost-effectiveness(sometime called benefit-cost)evaluations compare
eral levels or stages,with the ultimate goal generally understood to program benefits and costs,showing the relationship between the
be the increased adoption of energy efficiency goods and services in value of a program's benefits and the costs incurred to achieve those
the general market leading to energy savings. Energy savings are the benefits.The findings help judge whether to retain,revise,or eliminate
ultimate goal of programs seeking to cause market effects(i.e.,the program elements and provide feedback on whether efficiency is an
intended long-term outcome).The following list suggests a hierarchy effective investment,compared with energy supply options.Cost-
of precursors to that goal: effectiveness evaluation is also often a key component of the evalua-
• Early Acceptance:proliferation of models and manufacturers, tion process for programs using public or utility customer funds.
purchase by frontrunners and enthusiasts In 1983,California's Standard Practice for Cost-Benefit Analysis of
• Take-off Phase:customer awareness,visibility on shelves and Conservation and Load Management Programs manual(SPM)devel-
in inventories, perceptible levels of market share in the supply oped five cost-effectiveness tests for evaluating energy efficiency
channels programs.These approaches,with minor updates,continue to be
• Maturity:all major competitors offer energy efficient models; used today and are the principal approaches used for evaluating
codes and standards include energy efficient models energy efficiency programs across the United States.109
• Energy Savings:energy savings attributable to the program The five tests vary in terms of(1)perspectives(project participants,
are associated with acceleration of these developments. ratepayers,utilities,or society),(2)their applicability to different
In general,the achievement of goals at each of the higher levels program types,(3)the cost and benefit elements included in the
of the hierarchy requires accomplishments at the lower levels.As calculation,(4)the methods by which the cost and benefit elements
a result,tracking goals at each stage not only provides feedback are computed,and(5)the uses of the results. Most regulated utility
on performance with respect to that goal itself,but also provides energy efficiency programs use one or more versions of these tests,
evidence that effects at the next-higher levels can be attributed to sometimes with variations unique to the requirements of a particular
the program. regulatory commission.Definitions of these tests(paraphrased from
the SPM)are as follows on page 144:
Goals will typically be set and tracked for different time frames and • Total Resource Cost(TRC)Test.The TRC test measures the net
for different purposes.While energy savings are the ultimate market costs of a demand-side management program as a resource
effects goal,in most cases,savings cannot be measured meaningfully option based on the total costs of the program,including
December 2012 www.seeaction.energy.gov B-4

TABLE B.3:Approaches for Assessing Market Effects, Including Attribution
BASIC SOURCE/RELATIVE ADVANTAGES LIMITATIONS
SURVEYS OF CUSTOMER PURCHASES • Limited accuracy on key details:exact number,timing,
• Can be deployed quickly,relatively inexpensively,and efficiency rating of purchases
repeatedly over extended time frames • Non-response bias a problem,particularly in early stages
• Can be deployed in program and non-program areas of market development
• Generally produces reliable data on number of • Difficult to validate results in absence of some comparison
purchases/adoptions to sales or program volumes
SURVEYS OF SUPPLY-SIDE ACTORS • Difficult to build measures of sales volume—may need to
• Taps into close knowledge of local markets be content with estimates of market share
• Respondents sufficiently knowledgeable to provide • In many jurisdictions,population available to be sampled
accurate information on product features is small
• Difficult to validate results in absence of some comparison
to sales or program volumes
SHIPMENT AND SALES DATA • Requires negotiated cooperation of manufacturers and
• Conceptually,the most accurate and detailed measure retailers;risk of dropouts
of adoption:quantity,efficiency,timing • Difficult to obtain coverage of all sectors,time periods,and
regions(and may be costly)
• Quality control is difficult
CUSTOMER-REPORTED FREE RIDERSHIP AND SPILLOVER • For non-participants, requires that customers be aware
• Can be deployed quickly,relatively inexpensively,and of the program and able to judge its impact on adoption
repeatedly over extended time frames decisions
• Can probe adoption process and decisions
• Consistent with current Performance Earnings Basis(PEB)
methods now in force in California
CROSS-SECTIONAL METHODS • Increasingly difficult to find non-program areas
• Closest to conventional social science research methods; • Difficult to verify comparability of non-program areas
intuitively satisfying • Appears to be effective only in time-limited periods
• Data provide insight into exogenous factors,working of . Logistically demanding and time consuming
market beyond program boundary
EXPERT JUDGING • Not a statistical estimation process
• Focuses insights from experienced market participants • Difficult to identify and account for factors affecting
and observers individual judgments
• Results can be expressed in terms of net adoptions
• In some cases,can be deployed fairly rapidly
Source:Rosenberg,M.(June 2010)"Market Effects and Market Transformation:Their Role in Program Design and Evaluation."EPA EM&V Webinar.www.emvwebinar.org.
B-5 www.seeaction.energy.gov December 2012

both the participants'and the utility's costs. It combines the calculation.Results are reported either in net present value(NPV)
perspectives of participants and non-participants,which is why dollars(method by difference)or as a ratio(i.e.,benefits/costs).Table
it is also often called an"all ratepayers" perspective.The TRC B.4 outlines the basic approach underlying cost-effectiveness tests.
ratio equals the benefits of the program,in terms of value of
energy and demand saved,divided by the net costs.The ratio Each of the tests provides a different kind of information about
is usually calculated on a lifecycle basis,considering savings the impacts of energy efficiency programs from different vantage
and costs that accrue over the lifetime of installed energy points in the energy system.On its own,each test provides a single
efficiency equipment or systems.This is a commonly applied stakeholder perspective.Together,multiple tests provide a com-
cost-effectiveness test. prehensive approach to answering key questions:"Is the program
• Program Administrator Cost Test(PACT).The PACT mea- effective overall?""Is it balanced?"`Are some costs or incentives too
sures the net costs of a demand-side management program high or too low?""What is the effect on rates?""What adjustments
as a resource option based on the costs incurred by the are needed to improve the alignment?"
program administrator(often a utility,though it can be any Overall,the results of all five cost-effectiveness tests provide a more
organization),excluding any net costs incurred by the partici- complete picture than the use of any one test alone.The TRC and SCT
pant.The benefits are the same as the TRC benefits(energy cost tests help to answer whether energy efficiency is cost-effective
and demand savings value).The PACT is also a commonly overall.The PCT,PACT,and RIM help to answer whether the selection
applied test. of measures and design of the program is balanced from participant,
• Participant Cost Test(PCT).The PCT assesses cost effectiveness utility,and non-participant perspectives,respectively.Looking at the
from the participating consumer's perspective by calculating cost-effectiveness tests together helps to characterize the attributes
the quantifiable benefits and costs to the consumer of par- of a program or measure to enable decision making,to determine
ticipating in a program. Because many consumers do not base whether some measures or programs are too costly,whether some
their decision to participate entirely on quantifiable criteria, costs or incentives are too high or too low,and what adjustments
this test is not necessarily a complete measure of all the need to be made to improve distribution of costs and benefits among
benefits and costs a participant perceives. stakeholders.The scope of the benefit and cost components included
• Societal Cost Test(SCT).The SCT,a modified version of the TRC, in each test is summarized in Table B.5 and Table B.6.
adopts a societal rather than a utility service area perspective.The
primary difference between the societal and TRC tests is that,to The broad categories of costs and benefits included in each cost-
calculate lifecycle costs and benefits,the societal test accounts effectiveness test are consistent across all regions of the country
for externalities(e.g.,environmental benefits),excludes tax credit and applications. However,the specific components included in
benefits,and uses a(often lower)societal discount rate. each test may vary across different regions, market structures,and
• Ratepayer Impact Measure(RIM)Test.The RIM test only utility types. For example,transmission and distribution investment
may be considered deferrable through energy efficiency in some
applies to utility programs. It examines the potential impact
that the energy efficiency program has on rates overall.The net areas and not in others. Likewise,the TRC and SCT may consider just
benefits are the avoided cost of energy(same as PACT).The natural gas or electricity resource savings in some cases, but also
net costs include the overhead and incentive costs(same as include co-benefits of other savings streams(such as water and fuel
PACT)but also include utility lost revenues from customer bill oil)in others.
savings. Historically, reliance on the RIM test has limited energy Also,for the SCT,how the"non-monetized benefits"in Tables B.5 and
efficiency investment,as it is the most restrictive of the five B.6 are determined is an evolving area.In particular,benefits that
cost-effectiveness tests. in the past could not be monetized(e.g.,air quality impacts)now
The basic structure of each cost-effectiveness test involves a calcula-
can be assigned monetary values and,in fact,need to be assigned
lion of the total benefits and the total costs in dollar terms from a
such values in order to be used in cost-effectiveness equations.Also,
certain vantage point to determine whether or not the overall ben-
non-energy benefits,which in the past might have been ignored,
efits exceed the costs.A test is positive if the benefit-to-cost ratio is are being shown to have significant value.These include economic
greater than one,and negative if it is less than one—with,of course, development and employment benefits as more money is spent on
local services and products because of the efficiency investments.
proper consideration of uncertainties in the inputs used in the
December 2012 www.seeaction.energy.gov B-6

TABLE B.4: The Five Principal Cost-Effectiveness Tests Used in Energy Efficiency
ACRONYMTEST • OF - •
Participant cost test PCT Will the participants benefit over the Comparison of costs and benefits of the
measure life? customer installing the measure
Program administrator PACT Will utility bills increase? Comparison of program administrator
cost test costs to supply-side resource costs
Ratepayer impact measure RIM Will utility rates increase? Comparison of administrator costs and
utility bill reductions to supply-side
resource costs
Total resource cost test TRC Will the total costs of energy in the Comparison of program administrator and
utility service territory decrease? customer costs to utility resource savings
Societal cost test SCT Is the utility,state,or nation better off Comparison of society's costs of energy
as a whole? efficiency to resource savings and non-
cash costs and benefits
California Public Utilities Commission(CPUC).(2001).California Standard Practice Manual:Economic Analysis of Demand-Side Programs and Projects.www.energy.ca.gov/greenbuilding/
documents/background/07-J CPUC STANDARD PRACTICE MANUAL.PDF.
National Action Plan for Energy Efficiency.(2008).Understanding Cost-Effectiveness of Energy Efficiency Programs:Best Practices,Technical Methods,and Emerging Issues for Policy-Makers.
Prepared by Energy and Environmental Economics,Inc.(E3)and Regulatory Assistance Project(RAP).www.epa.gov/cleonenergy/documents/suca/cost-effectiveness.pdf.
B-7 www.seeaction.energy.gov December 2012

TABLE B.5: Description of Benefits and Costs Included in Each Cost-Effectiveness Test
PCT Benefits and costs from the perspective of the customer installing the measure
• Incentive payments • Incremental equipment costs
• Bill savings • Incremental installation costs
• Applicable tax credits or incentives
i
PACT Perspective of utility,government agency,or third party implementing the program
• Energy-related costs avoided by the utility • Program overhead cost
• Capacity-related costs avoided by the utility, including generation, • Utility/program administrator incentive costs
transmission,and distribution • Utility/program administrator installation costs
RIM Impact of efficiency measure on non-participating ratepayers overall
• Energy-related costs avoided by the utility • Program overhead cost
• Capacity-related costs avoided by the utility, • Utility/program administrator incentive costs
including generation,transmission,and distribution • Utility/program administrator installation costs
• Lost revenue due to reduced energy bills
TRC Benefits and costs from the perspective of all utility customers(participants and non-participants)in the utility service territory
• Energy-related costs avoided by the utility • Program overhead costs
• Capacity-related costs avoided by the utility, including generation, • Program installation costs
transmission,and distribution • Incremental measure costs(weather paid by
• Additional resource savings(i.e.,gas and water if utility is electric) the customer or utility)
• Monetized environmental and non-energy benefits
(see Section 4.9)
• Applicable tax credits(see Section 6.4)
SCT Benefits and costs to all the utility service territory,state,or nation as a whole
• Energy-related costs avoided by the utility • Program overhead costs
• Capacity-related costs avoided by the utility,including generation, • Program installation costs
transmission,and distribution • Incremental measure costs(weather paid by
• Additional resource savings(i.e.,gas and water if utility is electric) the customer or utility)
• Non-monetized benefits(and costs)such as cleaner air or
health impacts
California Public Utilities Commission(CPUC).(2001).California Standard Practice Manual:Economic Analysis of Demand-Side Programs and Projects.
www.energy.ca.gov/greenbuilding/documents/background/07-J CPUC STANDARD_PRACTICE MANUAL.PDF.
National Action Plan for Energy Efficiency.(2008).Understanding Cost-Effectiveness of Energy Efficiency Programs:Best Practices,Technical Methods,and Emerging Issues for Policy-
Makers.Prepared by Energy and Environmental Economics,Inc.(E3)and Regulatory Assistance Project(RAP).www.epa.gov/cleanenergy/documents/suca/cost-effectiveness.pdf.
December 2012 www.seeaction.energy.gov B-8

TABLE B.6: Summary of Benefits and Costs Included in Each Cost-Effectiveness Test
PCT PACT RIM
Energy and capacity-related avoided costs Benefit Benefit I Benefit Benefit
Additional resource savings Benefit Benefit
Non-monetized benefits Benefit
i
Incremental equipment and installation costs Cost Cost
i
Program overhead costs Cost Cost Cost Cost
Incentive payments Benefit Cost Cost
LBill savings Benefit Cost
California Public Utilities Commission(CPUC).(2001).California Standard Practice Manual:Economic Analysis of Demand-Side Programs and Projects.
www.energy.co.gov/greenbuilding/documents/background/07-J CPUC STANDARD_PRACTICE MANUAL.PDF.
National Action Plan for Energy Efficiency.(2008).Understanding Cost-Effectiveness of Energy Efficiency Programs:Best Practices,Technical Methods,and Emerging Issues for Policy-
Makers.Prepared by Energy and Environmental Economics,Inc.(0)and Regulatory Assistance Project(RAP).www.epa.gov/cleanenergy/documents/suca/cost-effectiveness.pdf.
B.2 TOP-DOWN IMPACT EVALUATION estimating net program savings at the sector,state,regional,and
Top-down impact evaluation refers to methods that rely on aggregate national levels.The primary potential drawbacks of top-down evalu-
energy consumption data or per-unit energy consumption indices ation are the difficulty in attributing energy consumption changes
(e.g.,energy consumption per-unit of output or per capita)defined by
to specific energy efficiency policies and/or particular programs and
sector,utility service territory,state,region,or country as the starting actions.
point for determining energy savings.Top-down evaluation focuses A metric that can be considered the output of top-down evaluation
on the bottom line—reductions in energy use(and/or demand)for a is gross market savings.These are the energy savings resulting from
state,region,or utility service territory.This gives top-down evaluation energy efficiency programs,codes and standards,and naturally
a direct link to(1)demand forecasting and resource planning,and occurring adoption,and which have a long-lasting savings effect.
(2)emissions accounting and forecasting,as used for greenhouse gas Such gross market savings sometimes do not include temporary
mitigation goals. reductions in energy use from changes in weather,income,energy
Figure B.1 compares the top-down with the bottom-up impact prices,and other structural economic changes,such as in industry
evaluation approaches that are discussed in the body of this guide. composition.Figure B.2 shows a graphical illustration of the concept
At present,virtually all energy efficiency program evaluations behind estimating gross market savings.110
conducted in the United States rely on bottom-up approaches. During the last two decades,many energy efficiency practitioners
Top-down approaches start from aggregate data,such as state-level and policymakers have expressed growing interest in the use of
data for energy consumption,and then attempt to correlate any top-down methods for documenting the system-wide impacts and
changes in energy consumption with measures of energy efficiency gross market savings of energy efficiency initiatives. Interest in
actions,such as expenditures or savings,using macro-economic top-down methods has grown from policymakers'and evaluation
models.The main advantages of top-down evaluation methods over researchers'concerns that bottom-up evaluations have not properly
bottom-up methods are their potentially lower evaluation costs due accounted for effects of free ridership,spillover,and energy effi-
to relatively modest data requirements and the potential for directly ciency measure interactions—particularly in large program portfolios
and in situations where energy consumer funds are used,such as in
B-9 www.seeaction.energy.gov December 2012

utility-sponsored efficiency programs.Top-down evaluations should FIGURE B.1: Comparison of bottom-up versus
also be less expensive to implement than bottom-up evaluations. top-down evaluation
Thus,research on top-down evaluation has been directed largely
toward estimation of energy consumer-funded energy efficiency
program savings,both the gross market energy savings and the Cl) Cl)
N
portion attributable to the programs being evaluated. J
Q .I. 1 J
Top-down energy efficiency evaluation methods are generally Z Z
less developed than bottom-up methods in the energy efficiency Z Q
field. However,at time of the publication of this guide,two macro- d
consumption(top-down)pilot studies are under way.Sponsored by 0
the California Public Utilities Commission(CPUC),these studies are d O
attempts to test the effectiveness and reliability of different top- O ~
H
down evaluation approaches and to determine whether they can ~ , , Om
be applied consistently to project market gross savings and to
attribute savings to utility-sponsored energy efficiency investments.
The results of these pilots can be found at the CALMAC website
(www.calmac.org)and the CPUC website(www.cpuc.ca.gov/PUC/
energy/Energy+Efficiency).
FIGURE B.2: Graphical illustration of estimation of market gross savings
10,000
9,000 Observed
consumption
8,000
Counterfactual
7,000 consumption
GWh
6,000
5,000
4,000
3,000
1987 1989 1991 1993 1995 1997 1999 2001 2003 2005 2007 2009
Source:Stewart J.;Haeri,M.H.(July 2011).Critical Review and Recommendations on Top-Down Evaluation.White paper.The Cadmus Group,Inc.Prepared for the California Public
Utilities Commission(CPUQ.
December 2012 www.seeaction.energy.gov B-10

Appendix B: Notes
105 Eto,J.; Prahl, R.;Schegel,J. (1996).A Scoping Study on Energy-
Efficiency Market Transformation by California Utility DSM Programs.
Lawrence Berkeley National Laboratory.http://eetd.lbl.gov/ea/ems/
reports/39058.pdf.
106 Rosenberg,M.; Hoefgen,L.(March 2009).Market Effects and
Market Transformation:Their Role in Energy Efficiency Program
Design and Evaluation. Prepared for California Institute for Energy
and Environment(CIEE)by KEMA,Inc.; Nexus Market Research.
http://uc-ciee.org/downloads/mrkt_effts_wp.pdf.
107 Much of this subsection is taken(in some cases word for
word)from the National Action Plan for Energy Efficiency. (2008).
Understanding Cost-Effectiveness of Energy Efficiency Programs:Best
Practices, Technical Methods,and Emerging Issues for Policy-Makers.
Prepared by Energy and Environmental Economics, Inc. (E3)and
Regulatory Assistance Project(RAP).www.epa.gov/cleanenergy/
documents/suca/cost-effectiveness.pdf.
108 Another recent summary report of cost-effectiveness tests is:
Woolf,T.;Malone, E.;Takahashi, K.;Steinhurst,W.(July 23,2012).
Best Practices in Energy Efficiency Program Screening:How to Ensure
that the Value of Energy Efficiency is Properly Accounted For.
Synapse Energy Economics,Inc. Prepared for National Home
Performance Council.www.synapse-energy.com/Downloads/
SynapseReport.2012-07.N HPC.EE-Program-Screening.12-040.pdf.
109 California Public Utilities Commission(CPUC).(2001).California
Standard Practice Manual:Economic Analysis of Demand-Side
Programs and Projects.www.energy.ca.gov/greenbuilding/docu-
ments/background/07-J_CPUC_STAN DARD_PRACTICE_MAN UAL.
PDF.See also www.cpuc.Ca.gov/PUC/energy/Energy+Efficiency/
EM+and+V for the 2007 SPM Clarification Memo.
110 Stewart,J.; Haeri,M.H.(July 2011).Critical Review and
Recommendations on Top-Down Evaluation.White paper.The
Cadmus Group,Inc. Prepared for the California Public Utilities
Commission(CPUC).
B-11 www.seeaction.energy.gov December 2012

Appendix C: Resources
This appendix provides a listing of references that provide a body Consortium for Energy Efficiency EM&V Resources:www.ceel.org/
of knowledge developed over the last several decades of energy eval/eval-res.php3.
efficiency program implementation and evaluation.They can be
considered the current primary resources for energy efficiency Efficiency Valuation Organization(EVO)International Performance
program impact evaluation and project measurement and verifica- Measurement and Verification Protocol(IPMVP):www.evo-world.org.
tion(M&V),and thus the basis for the definitions,approaches,and European Union Energy Efficiency Directive,measurement,
issues adopted and explained in this guide. In addition,throughout monitoring,and evaluation:www.evaluate-energy-savings.eu/
the guide there are numerous documents referenced in endnotes— emeees/en/home/index.php.
see individual guide sections for these references,as well as the
references section that follows this appendix. Federal Energy Management Program M&V:http://ateam.lbl.gov/mv.
C.1 BACKGROUND International Energy Program Evaluation Conference:www.iepec.org/
The information in this guide is documented in numerous guides, IEPECHome.htm?programsabstracts.htm.
protocols,papers,and reports. From a historical perspective,many Northeast Energy Efficiency Partnerships EM&V Forum:
of the basic references on energy and energy efficiency impact http://neep.org/emv-forum.
evaluations were written in the 1980s and 1990s. Unfortunately,
most of the early reference documents are not easily available to the Northwest Energy Efficiency Alliance evaluation:www.nwalliance.org/
general public(i.e.,they are not posted on the Web).However,here resea rch/eva I uation reports.aspx.
are three reference documents in the public domain that provide a State and Local Energy Efficiency Action Network Resources:
historical perspective and solid fundamentals:
www.seeaction.energy.gov/resources.html.
• Hirst, E.; Reed,J.eds. (1991).Handbook of Evaluation of
Utility DSM Programs.ORNL/CON-336. Prepared for Oak Ridge U.S. Environmental Protection Agency EM&V Webinar Series:
National Laboratory. www.emvwebinar.org.
• Khawaja, M.S.; Mulholland,C.;Thayer,J.;Smith, K.(2008). C.2.2 Select Impact Evaluation Resources
Guidebook for Energy Efficiency Program Evaluation California Institute for Energy and Environment.(2009).Energy
Measurement& Verification. 1016083. Palo Alto,CA: Electric Efficiency O Evaluation Training
Power Research Institute(EPRI). Opportunities.http://uc-ciee.org/
planning-evol uation/7/342/105/nested.
• Violette, D. M.(1995).Evaluation, Verification, and
Performance Measurement of Energy Efficiency Programmes. California Public Utilities Commission.(2001).California Standard
Prepared for International Energy Agency. Practice Manual:Economic Analysis of Demand-Side Programs
and Projects.www.energy.ca.gov/greenbuilding/documents/
RE PRIMARY IMPACT EVALUATION background/07-J_CPUC_STANDARD_PRACTICE_MANUAL.PDF.
RESOURCES See also www.cpuc.ca.gov/PUC/energy/Energy+Efficiency/
C.2.1 General Efficiency Evaluation Resource EM+and+V for the 2007SPM Clarification Memo.
Websites
American Council for an Energy-Efficient Economy California Public Utilities Commission.(2006).California Energy
(ACEEE)publications:www.aceee.org/publications. Efficiency Evaluation Protocols:Technical,Methodological,and
Reporting Requirements for Evaluation Professionals. Prepared
California Institute of Energy and Environment, Planning and by The TecMarket Works Team.www.calmac.org/events/
Evaluation Library:http://uc-ciee.org/planning-evaluation/7/ Eva IuatorsProtocols_Final_AdoptedviaRuIing_06-19-2006.pdf.
Ibrsearch.
California Measurement Advisory Council(CALMAC): California Public Utilities Commission.(2004).California Evaluation
www.calmac.org. Framework(2004).Prepared by The TecMarket Works
Team.www.calmac.org/publications/California_Evaluation_
Framework_J une_2004.pdf.
December 2012 www.seeaction.energy.gov C-1

Messenger, M.; Bharvirkar, R.;Golemboski,B.;Goldman,C.A.; C,2,3 U.S. Environmental Protection Agency
Schiller,S.R.(April 2010).Review of Evaluation,Measurement and Non-Energy Benefits and Avoided Emissions
Verification Approaches Used to Estimate the Load Impacts and Calculation References
Effectiveness of Energy Efficiency Programs. Lawrence Berkeley Assessing the Multiple Benefits of Clean Energy
National Laboratory. Report LBNL-3277E.http://emp.lbl.gov/ This document helps state energy,environmental,and economic
sites/all/files/Ibnl-3277e.pdf. policymakers identify and quantify the benefits of clean energy,
including the energy,environmental—specifically greenhouse gas,
State and Local Energy Efficiency Action Network.(April 2011). air,and health—and economic benefits of clean energy.It provides
National Energy Efficiency Evaluation,Measurement,and an analytical framework that states can use to estimate those
Verification(EM&V)Standard:Scoping Study of Issues and benefits during the development,implementation,and evaluation of
Implementation Requirements. Prepared by Schiller,S.R.; clean energy policies and programs. http://epa.gov/statelocalclimate/
Goldman,C.A.;Galawish,E.; LBNL Environmental Energy resources/benefits.html.
Technologies Division.wwwl.eere.energy.gov/seeaction/pdfs/
emvstandard_scopingstudy.pdf. Co-Benefits Risk Assessment(COBRA)Screening Model
EPA's Co-Benefits Risk Assessment(COBRA)screening model is a free
State and Local Energy Efficiency Action Network.(June 2011). tool that helps state and local governments estimate and map the
Scoping Study to Evaluate Feasibility of National Databases for air quality, human health,and related economic benefits(excluding
EM&V Documents and Measure Savings. Prepared by Jayaweera, energy cost savings)of clean energy policies or programs.
T.; Haeri, H.; Lee,A.; Bergen,S.; Kan,C.;Velonis,A.;Gurin,C.;
Visser,M.;Grant,A.; Buckman,A.;The Cadmus Group Inc. Roadmap for Incorporating Energy Efficiency/Renewable Energy
wwwl.eere.energy.gov/seeaction/pdfs/emvscoping_ Policies and Programs into State and Tribal Implementation Plans
data basefeasibility.pdf. The EPA Roadmap reduces the barriers for state,tribal,and local
agencies to incorporate energy efficiency and renewable energy(EE/
State and Local Energy Efficiency Action Network.(May 2012). RE)policies and programs in state and tribal implementation plans
Evaluation,Measurement,and Verification(EM&V)of Residential by clarifying existing EPA guidance and providing new and detailed
Behavior-Based Energy Efficiency Programs:Issues and information.The goal of this document is to facilitate the use of
Recommendations. Prepared by Todd,A.;Stuart,E.;Schiller,S.; energy efficiency and renewable energy emissions reduction strate-
Goldman,C.;Lawrence Berkeley National Laboratory. gies in air quality plans.www.epa.gov/airquality/eere/manual.html.
wwwl.eere.energy.gov/seeaction/pdfs/emv_behaviorbased_
eeprograms.pdf. Projected Impacts of Existing State Energy Efficiency and
Renewable Energy Policies
U.S.Department of Energy.(2006).EERE Guide for Managing General These EPA methods and projected energy impacts may be useful to
Program Evaluation Studies. Prepared by H. Barnes,Lockheed states preparing State Implementation Plan(SIP)submittals to meet
Martin Aspen;Gretchen Jordan,Sandia National Laboratories. the National Ambient Air Quality Standards(NAAQS)for ozone and
wwwl.eere.energy.gov/analysis/pdfs/evl_mg_app.pdf. particulate matter.www.epa.gov/statelocalclimate/state/
statepolicies.html.
U.S. Department of Energy.(2007).Impact Evaluation Framework
for Technology Deployment Programs. Prepared by J.Reed, Emissions&Generation Resource Integrated Database(eGRID)
Innovologie,LLC;G.Jordan,Sandia National Laboratories;E.Vine, eGRID is a comprehensive source of data on the environmental
Lawrence Berkeley National Laboratory.wwwl.eere.energy.gov/ characteristics of almost all electric power generated in the United
analysis/pdfs/impact_framework_tech_deploy_2007_overview.pdf. States.These include air emissions of nitrogen oxides,sulfur
dioxide,carbon dioxide,and many other attributes.www.epa.gov/
U.S. Environmental Protection Agency.(2012). The Roadmap for cleanenergy/energy-resources/egrid/index.html.
Incorporating Energy Efficiency/Renewable Energy Policies and
Programs into State and Tribal Implementation Plans. (See
Appendix I of the roadmap).www.epa.gov/airquality/eere.
C-2 www.seeaction.energy.gov December 2012

Power Plant Emissions Calculator(P-PEC) Hourly Marginal Emissions Tool
P-PEC is a simplified tool that uses eGRID"non-baseload"emissions This tool is a statistical dispatch simulator that predicts the hourly
factors to help states quickly estimate the magnitude of emission changes in generation and air emissions at electric generating units
reductions from EE/RE for each power plant within a region,and (EGUs)resulting from EE/RE policies and programs.Contact Robyn
to understand potential emission reductions within a county or DeYoung at EPA:[e-mail]
air quality nonattainment area.Contact Robyn DeYoung at EPA:
[e-mail]
C.2.4 Technical Reference Manual (TRMs) Resources
TABLE CA: United States Technical Reference Manuals
RESOURCE NAME INFORMATION® ADMINISTRATORINCLUDED
www.energystar.gov/index. Online Ex ante savings
National ENERGY STAR° cfm?c=products. Calculators based on Agency
pr_find_es_products algorithms
Regional Technical Ex ante savings
Regional— www.nwcouncil.org/energy/rtf/ Online Advisory
Forum(RTF)Deemed based on
Northwest Measures measures/Default.asp Database algorithms Committee
http://neep.org/uploads/
Regional— EMV%20Forum/EMV%2OProducts/ Algorithms and Nonprofit
Mid-Atlantic Mid Atlantic TRIM PDF Mid%20Atlantic%20TRM_ ex ante savings Organization
V1%202_FINAL.pdf
Arkansas Deemed www.aepefficiency.com/
Arkansas Savings Quick Start oklahoma/ci/downloads/ PDF Algorithms Public Utility
Programs Deemed_Savings_Report.pdf
DEER Database for Software
California Energy-Efficient www.deeresources.com Ex ante savings State Commission
resources Program
Connecticut Light&
Power and United www.ctenergyinfo.com/2012 CT
Connecticut Illuminating Company Program Savings Documentation PDF Algorithms and ex public Utility
Program Savings FINAL.pdf ante savings
Documentation
Hawaii Energy Efficiency www.hawaiienergy.com/media/ Algorithms and ex
Hawaii assets/AnnualReportAttachments- PDF State Commission
Program TRIMALL.pdf ante savings
www.efficiencymaine.com/docs/
Efficiency Maine TRIM board_meeting_documents/ PDF Algorithms and Trust
—Commercial Maine-Commercial-TRM-8-31- ex ante savings
2010-Final.pdf
Maine
www.efficiencymaine.com/docs/
Efficiency Maine TRIM board_meeting_documents/ PDF
—Residential Maine-Residential-TRM-02-04-09.
pdf
Massachusetts www.ma-eeac.org/docs/
Statewide TRIM for 2012 MTM Files/Oct 28,2011 DPU Algorithms and
Massachusetts Estimating Filings/WMECo/13-WMECO PDF ex ante savings Agency
Savings from Energy Exhibit H MA TRM_2012 PLAN_
Efficiency Measures FINAL.pdf
December 2012 www.seeaction.energy.gov C-3

INFORMATION
7Mric
0
. . - •-
INCLUDEDwww.michigan.gov/mpsc/Michigan Energy Excel
Measures Database Database an 0,1607,7-159-52495_ Ex ante savings State Commission
55129---,00.html
New Jersey Clean Energy www.njcleanenergy.com/files/
Algorithms and
New Jersey Program Protocols to file/Library/NJ Protocols Revisions PDF Agency
Measure Resource Savings 7-21-11_Clean.pdf ex ante savings
New York Standard
Approach for Estimating
New York Energy Savings from http://efile.mpsc.state.mi.us/efile/ PDF Algorithms and Agency
docs/16671/0026.pdf ex ante savings
Energy Efficiency
Programs
Ohio Ohio TRM Online Algorithms and www.ohiotrm.org State Commission
Database ex ante savings
www.puc.state.pa.us/electric/ Algorithms and
Pennsylvania Pennsylvania TRM DOC State Commission
Actl29/TRM.aspx ex ante savings
Deemed Savings,
www.entergy-texas.com/content/
Energy_Efficiency/documents/ Algorithms and
Texas Installation,and PDF State Commission
Efficiency Standards
Deemed—
Savings_Measures_ ex ante savings
List.pdf
Efficiency Vermont
www.veic.org/Libraries/Resumes/ Algorithms and Nonprofit
Vermont Technical Reference User PDF
Manual Tech Manua lEVT.sflb.ashx ex ante savings Organization
Focus on Energy www.focusonenergy.com/files/
Evaluation Business Document—Management_System/ Algorithms and
Wisconsin PDF State Commission
Programs: Deemed Evaluation/bpdeemedsavings- ex ante savings
Savings Manual V1.0 manuavl0_evaluationreport.pdf
Source:State and Local Energy Efficiency Action Network.(June 2011).Scoping Study to Evaluate Feasibility of National Databases for EM&V Documents and Measure Savings.
Prepared byJayaweera,T.;Hoed,H.;Lee,A.;Bergen,S.;Kan,C.;Velonis,A.;Gunn,C.;Visser,M.;Grant A.;Buckman,A.;The Cadmus Group Inc.
wwwl.eere.energy.govlseeaction/pdfs/emvscoping databasefeasibility.pdf.Tables are current as of Summer 2012.
C-4 www.seeaction.energy.gov December 2012

C.3 MEASUREMENT AND VERIFICATION ASHRAE Performance Measurement Protocols(PMP)for
RESOURCES Commercial Buildings.These provide a standardized,con-
Several M&V resource documents used in the development of this sistent set of protocols for facilitating the comparison of
guide are available via the internet and are presented in this section; measured energy,water,and indoor quality performance of
they can be considered the current primary resources for energy commercial buildings.www.ashrae.org.
efficiency project M&V. • International Standards Organization(ISO).The International
• International Performance Measurement and Verification Standards Organization entered the world of energy manage-
Protocol(IPMVP).The IPMVP provides an overview of current ment with the release of ISO 50001-Energy Management
best practices for verifying results of energy efficiency,water, Systems.This protocol includes the following:
and renewable energy projects in commercial and industrial — ISO 50001:2011 specifies requirements for establishing,
facilities. Internationally, it is the most recognized M&V implementing, maintaining,and improving an energy
protocol for demand-side energy activities.The IPMVP was management system,the purpose of which is to enable an
developed with DOE sponsorship and is currently managed organization to follow a systematic approach in achieving
by the nonprofit Efficiency Valuation Organization,which continual improvement of energy performance,including
continually maintains and updates the IPMVP. energy efficiency,energy use,and consumption.
The IPMVP provides a framework and definitions that can — ISO 50001:2011 specifies requirements applicable to
help practitioners develop M&V plans for their projects. It energy use and consumption,including measurement,
includes guidance on best practices for determining savings documentation and reporting,design,and procurement
from efficiency projects.It is not a"cookbook"of how to practices for equipment,systems, processes,and personnel
perform M&V for specific projects;rather,it provides guid- that contribute to energy performance.
ance and key concepts that are used in the United States and — ISO 50001:2011 applies to all variables affecting energy
internationally.The IPMVP is probably best known for defining performance that can be monitored and influenced by the
four M&V options for energy efficiency projects.These options organization. ISO 50001:2011 does not prescribe specific
(A,B,C,and D),presented in Chapter 4,differentiate the most performance criteria with respect to energy.
common approaches for M&V. Reference:Efficiency Valuation — Subsequent to the release of 50001, ISO created a
Organization(2010).International Performance Measurement technical committee(TC242)to support the deployment of
and Verification Protocol.www.evo-world.org. ISO 50001. In January 2011,the ISO Technical Management
• FEMP M&V Guidelines.The purpose of this document is to Board announced the creation of another committee to
provide guidelines and methods for documenting and verify- create standards on measuring and verifying savings—TC
ing the savings associated with federal agency performance 257 Energy Savings.Any standards originating from TC 242,
contracts. It contains procedures and guidelines for quantifying TC 257,or the joint working group are expected to provide
the savings resulting from energy efficiency equipment,water broad guidance on M&V principles in 2012 or 2013.
conservation,improved operations and maintenance, renew-
able energy,and cogeneration projects.Along with the FEMP
M&V Guidelines are several useful companion documents.
http://mnv.lbl.gov/keyMnVDocs.
• ASHRAE Guideline 14 Measurement of Energy and Demand
Savings.ASHRAE is the professional engineering society that
has been the most involved in writing guidelines and standards
associated with energy efficiency.Compared with the FEMP
M&V Guidelines and the IPMVP,Guideline 14 is a more detailed
technical document that addresses the analyses,statistics,and
physical measurement of energy use for determining energy
savings. Reference:American Society of Heating,Refrigerating,
and Air-Conditioning Engineers(2002).A new version is
expected to be available in 2013.www.ashrae.org.
December 2012 www.seeaction.energy.gov C-5

References
"Acid Rain Program."(2012).U.S.Environmental Protection Agency. California Public Utilities Commission(CPUC).(2007).Guidelines for
www.epa.gov/airmarkets/progsregs/arp. Estimating Net-To-Gross Ratios Using the Self-Report Approaches.
CPUC Energy Division Master Evaluation Contractor Team.
"Air Markets Program Data."(2012).U.S. Environmental Protection ftp://ftp.cpuc.ca.gov/puc/energy/electric/energy+efficiency/
Agency. http://ampd.epa.gov/ampd/. ee+workshops/self repo rtguidelinesd eta iled_v20.pdf.
American Council for an Energy-Efficient Economy(ACEEE). California Public Utilities Commission(CPUC).(April 9,2010).
(October 2011).State Energy Efficiency Resource Standard(EERS) Codes&Standards(C&S)Programs Impact Evaluation:
Activity. http://aceee.org/files/pdf/policy-brief/.State EERS California Investor Owned Utilities'Codes and Standards Program
Summary October 2011.pdf. Evaluation for Program Years 2006-2008. CALMAC Study ID:
CP00030.06 Final Evaluation Report.Prepared by KEMA, Inc.;
American Society of Heating, Refrigerating and Air-Conditioning The Cadmus Group,Inc.; Itron, Inc.;Nexus Market Research,
Engineers(ASHRAE).(2002).Guideline 14:Measurement of Inc.;and ENRG,Inc.for the CPUC Energy Division.
Demand and Energy Savings.www.ashrae.org. www.calmac.org/publications/Codes_Standards—Vol-111—
Fi na I Eva I uation ReportU pdated-04122010.pdf.
American Society of Heating, Refrigerating and Air-Conditioning
Engineers(ASHRAE).(2010).Performance Measurement "Clean Air Interstate Rule."(2005). U.S. Environmental Protection
Protocols for Commercial Buildings.www.ashrae.org/ Agency.www.epa.gov/cair.
standards-research--technology/special--project-activities.
"Compilation of Air Pollutant Emission Factors."(2011). U.S.
Barbose,G.; Billingsley; M.;Goldman,C.;Hoffman,I.;Schlegel, Environmental Protection Agency. EPA AP-42.www.epa.gov/ttn/
J.(August 2012)."On a Rising Tide:The Future of U.S. Utility chief/ap42.
Customer-Funded Energy Efficiency Programs."2012 ACEEE
Summer Study Proceedings,August 12-17,2012, Pacific Grove, Consensus Report on the EmPower Maryland EM&V Process.(June
California.Washington,D.C.:American Council for an Energy- 24,2009).www.raponline.org/document/download/id/551.
Efficient Economy(ACEEE).LBNL-5755E. http://www.aceee.org/
files/proceedings/2012/data/papers/0193-000173.pdf. Efficiency Valuation Organization(EVO). International Performance
Measurement and Verification Protocol(IPMVP).(multiple
"Building Energy Codes Program."(2012). U.S. Department of Energy. dates).www.evo-world.org.
www.energycodes.gov.
"eGRID."(2012). U.S.Environmental Protection Agency.
"California Greenhouse Gas(GHG)Cap-and-Trade Program."(2012). www.epa.gov/cleanenergy/energy-resources/egrid/index.html.
California Air Resources Board(CARB).www.arb.ca.gov/cc/
capandtrade/capandtrade.htm. Energy&Resources Solutions,Inc.(ERS);Left Fork Energy;West
Hill Energy&Computing; Megdal&Associates,LLC.;
California Public Utilities Commission(CPUC).(2001).California Opinion Dynamics Corporation.(June 2010).Report
Standard Practice Manual:Economic Analysis of Demand-Side on Impact Assessments of Two NYSERDA Natural Gas
Programs and Projects.www.energy.ca.gov/greenbuilding/ Efficiency Programs:Con Edison Natural Gas Efficiency
documents/background/07-J_CPUC_STANDARD_PRACTICE_ Program&National Grid Low Income Natural Gas
MANUAL.PDF.See also http://www.cpuc.ca.gov/PUC/energy/ Efficiency Program. Prepared for the New York State
Energy+Efficiency/EM+and+V/. Energy Research and Development Authority(NYSERDA).
http://documents.dps.ny.gov/public/Common/ViewDoc.
California Public Utilities Commission(CPUC).(April 2006). aspx?DocRefld={CFCD7DE1-8B68-46CC-B64B-83ACD072A4FD).
California Energy Efficiency Evaluation Protocols:Technical,
Methodological,and Reporting Requirements for Evaluation Eto,J.; Prahl,R.;Schegel,J.(1996).A Scoping Study on Energy-
Professionals.www.calmac.org/events/EvaluatorsProtocols_ Efficiency Market Transformation by California Utility DSM
Final_AdoptedviaRuling_06-19-2006.pdf. Programs.Lawrence Berkeley National Laboratory.
http://eetd.lbl.gov/ea/ems/reports/39058.pdf.
D-1 www.seeaction.energy.gov December 2012

Federal Energy Management Program(FEMP).(April 2008).M&V Intergovernmental Panel on Climate Change(IPCC).(2007).
Guidelines:Measurement and Verification for Federal Energy Pathways to a Low-Carbon Economy:Version 2 of the Global
Projects.wwwl.eere.energy.gov/femp/pdfs/mv_guidelines.pdf. Greenhouse Gas Abatement Cost Curve. Prepared by McKinsey
&Company.www.epa.gov/statelocalclimate/documents/pdf/
Federal Energy Regulatory Commission(FERC).(June 2010). mckinsey_summary_11-19-09.pdf.
National Action Plan on Demand Response. Docket No.AD09-10.
Prepared by FERC staff.www.ferc.gov/legal/staff-reports/ Kushler,M.; Nowak,S.;Witte,P.(February 2012).A National
06-17-10-demand-response.pdf. Survey of State Policies and Practices for the Evaluation of
Ratepayer-Funded Energy Efficiency Programs.American
"Global Warming Solutions Act:Assembly Bill 32."(2006).California Council for an Energy-Efficient Economy(ACEEE). Report
Air Resources Board(CARB).www.arb.ca.gov/cc/ab32/ab32.htm. Number U122.www.aceee.org/research-report/ul22.
Gottstein,M.;Schwartz,L.(2010).The Role of Forward Capacity Lazar,J.;Baldwin,X.(August 2011). Valuing the Contribution of
Markets in Increasing Demand-Side and Other Low-Carbon Energy Efficiency to Avoided Marginal Line Losses and Reserve
Resources Experience and Prospects. Regulatory Assistance Requirements. Regulatory Assistance Project(RAP).
Project(RAP).www.raponline.org/docs/RAP_Gottstein_ www.raponline.org/document/download/id/4537.
Schwa rtz_RoleofFCM_Experienceand Prospects2_2010_05_04.pdf.
Mahone,D.; Hall,N.(August 2010)."Pursuit of Aggressive Energy
Haeri, H.; Khawaja,M.S.(March 2012)."The Trouble With Savings Targets—Aligning Program Design and Evaluation."
Freeriders."Public Utilities Fortnightly.www.fortnightly.com/ 2010 ACEEE Summer Study Proceedings;August 15-20,2010,
fortnightly/2012/03/trouble-freeriders. Pacific Grove,California.Washington, D.C.:American Council
for an Energy-Efficient Economy. http://neep.org/uploads/
Hibbard,P.;Tierney,S.;Okie,A.; Darling,P.(November 2011). The EMV%20Forum/Steering%20Committee%20Notes/DMahone_
Economic Impacts of the Regional Greenhouse Gas Initiative on ACEEE_Paper_for_SC_Notes.pdf.
Ten Northeast and Mid-Atlantic States,Review of the Use of RGGI
Auction Proceeds from the First Three-Year Compliance Period. Mahone,D.(August 2011)."Free-Ridership as a Way to Kill
The Analysis Group.www.analysisgroup.com/uploadedfiles/ Programs—How Evaluation Policies Can Frustrate Efficiency
publishing/articles/economic_impact_rggi_report.pdf. Goals." Boston:International Energy Program Evaluation
Conference.www.iepec.org/20llPapersTOC/papers/054.pdf.
Hirst,E.;White, D.(1985).Indoor Temperature Changes After
Retrofit:Inferences Based on Electricity Billing Data for Messenger,M.; Bharvirkar, R.;Golemboski,B.;Goldman,C.A.;
Nonparticipants and Participants in the BPA Residential Schiller,S.R.(April 2010).Review of Evaluation,Measurement
Weatherization Program.Oak Ridge National Laboratory. and Verification Approaches Used to Estimate the Load Impacts
www.ornl.gov/info/reports/1985/3445600444954.pdf. and Effectiveness of Energy Efficiency Programs. Lawrence
Berkeley National Laboratory. Report LBNL-3277E.
Intergovernmental Panel on Climate Change(IPCC).(2006).2006 http://emp.lbl.gov/sites/all/files/Ibnl-3277e.pdf.
Guidelines for National Greenhouse Gas Inventories.
www.ipcc-nggip.iges.or.jp/public/2006gl/index.html. Mid-Atlantic Regional Air Management Association(MARAMA).
www.marama.org.
Intergovernmental Panel on Climate Change(IPCC).(2007).
Climate Change 2007:Synthesis Report.www.ipcc.ch/pdf/ National Action Plan for Energy Efficiency.(2007).Guide to Resource
assessment-report/ar4/syr/ar4_syr.pdf. Planning with Energy Efficiency. Prepared by Snuller Price et al.,
Energy and Environmental Economics,Inc.(E3).www.epa.gov/
Intergovernmental Panel on Climate Change(IPCC).(2007). cleanenergy/documents/suca/resource_planning.pdf.
Contribution of Working Group III to the Fourth Assessment
Report of the Intergovernmental Panel on Climate Change,2007. National Action Plan for Energy Efficiency.(2007).Model Energy
www.ipcc.ch/publications_and_data/ar4/wg3/en/contents.html. Efficiency Program Impact Evaluation Guide. Prepared by Steven
R.Schiller,Schiller Consulting,Inc.www.epa.gov/eeactionplan.
December 2012 www.seeaction.energy.gov D-2

National Action Plan for Energy Efficiency.(2008). Understanding Schiller,S.R.(August 2011). "Documenting Greenhouse Gas Emission
Cost-Effectiveness of Energy Efficiency Programs:Best Practices, Reductions from End-Use Efficiency Activities in the Kyoto
Technical Methods,and Emerging Issues for Policy-Makers. Protocol's Clean Development Mechanism Offset Program."
Prepared by Energy and Environmental Economics,Inc.(E3); 2011 International Energy Program Evaluation Conference
Regulatory Assistance Project(RAP).www.epa.gov/cleanenergy/ Proceedings,August 16-18,2011,Boston,Massachusetts.
documents/suca/cost-effectiveness.pdf. Madison,Wisconsin: International Energy Program Evaluation
Conference(IEPEC).www.iepec.org/20llPapersTOC/papers/
NMR Group, Inc.; Research Into Action,Inc.(November 2010)."Net 115.pdf#page=l.
Savings Scoping Paper."Submitted to Northeast Energy Efficiency
Partnerships:Evaluation, Measurement,and Verification Forum. Schiller,S.R.;Goldman,C.A.(August 2011)."Developing State
http://neep.org/uploads/EMV%20Forum/EMV%20Products/ and National Evaluation Infrastructures—Guidance for the
FINAL%20Net%20Savings%20Scoping%20Paper%2011-13-10.pdf. Challenges and Opportunities of EM&V."2011 International
Energy Program Evaluation Conference Proceedings,August
Nowak,S.;Kushler,M.;Sciortino,M.;York,D.;Witte,R(June 2011). 16-18,2011, Boston,Massachusetts. Madison,Wisconsin:
Energy Efficiency Resource Standards:State and Utility Strategies International Energy Program Evaluation Conference(IEPEC).
for Higher Energy Savings.American Council for an Energy-Efficient www.iepec.org/20llPapersTOC/papers/073.pdf#page=l.
Economy(ACEEE).Report Number U113.www.aceee.org/
research-report/u113. Schiller,S.R.; Prindle, B.;Cowart, R.;Rosenfeld,A.H.(2008)."Energy
Efficiency and Climate Change Mitigation Policy."2008ACEEE
Peters,J.;McRae,M.(August.2008)."Free-Ridership Measurement Summer Study Proceedings.August 17-22,2008,Pacific Grove,
Is Out of Sync With Program Logic...or,We've Got the California.Washington, DC:American Council for an Energy-
Structure Built, but What's Its Foundation?"2008ACEEE Efficient Economy(ACEEE).www.aceee.org/files/proceed-
Summer5tudyProceedings.August 17-22,2008,Pacific ings/2008/data/papers/8_306.pdf.
Grove,California.Washington, DC:American Council for an
Energy-Efficient Economy(ACEEE).www.aceee.org/files/ Skumatz, L.A.(1997)."Recognizing All Program Benefits: Estimating
proceed ings/2008/data/papers/5_491.pdf. the Non-Energy Benefits of PG&E's Venture Partners Pilot
Program(VPP)."Proceedings of the 1997 Energy Evaluation
"Promoting Air Quality and Climate Protection with Clean Energy Conference.August 27-29,1997,Chicago,Illinois.Madison,
and Energy Efficiency in the Metropolitan Washington Area." Wisconsin: International Energy Program Evaluation Conference
Metropolitan Washington Council of Governments(MWCOG). (IEPEC).www.iepec.org/1997PapersTOC/papers/033.pdf.
www.mwcog.org/environment/air/EERE/default.asp.
Skumatz, L.A.; Khawaya,M.;Colby,J.(2009).Lessons Learned and
Rathbun,P.;Sabo,C.;Zent, B.(2003).Standardized Methods for Next Steps in Energy Efficiency Measurement and Attribution:
Free-Ridership and Spillover Evaluation—Task 5 Final Report Energy Savings,Net to Gross,Non-Energy Benefits,and
(Revised). Prepared for National Grid, NSTAR Electric, Northeast Persistence of Energy Efficiency Behavior. Prepared for the
Utilities, Unitil,and Cape Light Compact.www.ceel.org/eval/ California Institute for Energy and Environment(CIEE)Behavior
db_pdf/297.pdf. and Energy Program. http://uc-ciee.org/downloads/EEM_A.pdf.
"Regional Greenhouse Gas Initiative."(2012).http://rggi.org/. Skumatz, L.A.; Khawaja,M.S.; Krop, R.(May 2010).Non-Energy
Benefits:Status,Findings,Next Steps,and Implications for Low
Rosenberg, M.;Hoefgen,L.(March 2009).Market Effects and Market Income Program Analyses in California. Revised Report.Prepared
Transformation:Their Role in Energy Efficiency Program Design for Brenda Gettig,Sempra Utilities.www.liob.org/docs/LIEE Non-
and Evaluation. Prepared for California Institute for Energy and Energy Benefits Revised Report.pdf.
Environment(CIEE)by KEMA, Inc.;Nexus Market Research.
http://uc-ciee.org/downloads/mrkt_effts—wp.pdf.
D-3 www.seeaction.energy.gov December 2012

State and Local Energy Efficiency Action Network.(April 2011). U.S. Environmental Protection Agency.(August 2011).Federal
National Energy Efficiency Evaluation,Measurement and Implementation Plans:Interstate Transport of Fine Particulate
Verification(EM&V)Standard:Scoping Study of Issues and Matter and Ozone and Correction of SIP Approvals.www.gao.gov/
Implementation Requirements. Prepared by Schiller,S.R.; prod ucts/C00127.40 CFR Parts 51,52,72,78,and 97[EPA—
Goldman,C.A.;Galawish,E.;LBNL Environmental Energy HQ—OAR-2009-0491;FRL-9436-8] RIN 2060—AP50.ACTION:
Technologies Division.wwwl.eere.energy.gov/seeaction/pdfs/ Final rule.
emvsta nda rd_scopingstudy.pdf.
Vine, E.; Hall,N.;Keating, K.M.;Kushler,M.; Prahl, R.(August 2011).
State and Local Energy Efficiency Action Network.(June 2011). "Emerging Evaluation Issues Revisited."2011 International Energy
Scoping Study to Evaluate Feasibility of National Databases for Program Evaluation Conference Proceedings,August 16-18,
EM&V Documents and Measure Savings. Prepared by Jayaweera, 2011,Boston,Massachusetts. Madison,Wisconsin:International
T.;Haeri, H.;Lee,A.; Bergen,S.;Kan,C.;Velonis,A.;Gurin,C.; Energy Program Evaluation Conference(IEPEC).www.iepec.org/
Visser, M.;Grant,A.;Buckman,A.;The Cadmus Group Inc. 2011PapersTOC/papers/096.pdf#page=l.
wwwl.eere.energy.gov/seeaction/pdfs/emvscoping_
databasefeasibility.pdf. Wallace,R;Forster,H.J.(2012).State of the Efficiency Program
Industry Budgets, Expenditures,and Impacts 2011.Consortium
State and Local Energy Efficiency Action Network.(May 2012). for Energy Efficiency.www.ceel.org/files/2011 CEE Annual
Evaluation,Measurement,and Verification(EM&V)of Residential Industry Report.pdf.
Behavior-Based Energy Efficiency Programs:Issues and
Recommendations. Prepared by Todd,A.;Stuart, E.;Schiller,S.; Woolf,T.;Malone, E.;Takahashi,K.;Steinhurst,W.(July 23,2012).
Goldman,C.; Lawrence Berkeley National Laboratory. Best Practices in Energy Efficiency Program Screening:How to
wwwl.eere.energy.gov/seeaction/pdfs/emv_behaviorbased_ Ensure that the Value of Energy Efficiency is Properly Accounted
eeprograms.pdf. For.Synapse Energy Economics, Inc. Prepared for National Home
Stewart,J.;Haeri, M.H.(July 2011).Critical Review and Performance Council.www.synapse-energy.com/Downloads/
Recommendations on Top-Down Evaluation. White paper. Syn a pse Re po rt.2012-07.N H PC.E E-Program-Sc re e n i n g.12-040.pd f.
The Cadmus Group,Inc. Prepared for the California Public World Resources Institute(WRI);World Business Council for
Utilities Commission(CPUC). Sustainable Development(WBCSD).(December 2005).
"Uniform Methods Project."(2012). U.S. Department of Energy. GHG Protocol for Project Accounting(Project Protocol).
wwwl.eere.energy.gov/deployment/ump.html. www•ghgprotocol.org/files/ghgp/ghg_project_protocol.pdf.
U.S. Environmental Protection Agency.(2004).Guidance on State World Resources Institute(WRI);World Business Council for
Implementation Plan(SIP)Credits for Emission Reductions from Sustainable Development(WBCSD).(August 2007).Protocol
Electric-Sector Energy Efficiency and Renewable Energy Measures. Guidelines for Quantifying GHG Reductions from Grid-Connected
www.epa.gov/ttn/oarpg/t1/memoranda/ereseerem_gd.pdf. Electricity Projects.www.wri.org/publication/guidelines-quantify-
ing-ghg-reductions-grid-connected-electricity-projects.
U.S. Environmental Protection Agency.(September 2005).(Draft).
State Set-Aside Programs for Energy Efficiency and Renewable Zabin,C.(2011).California Workforce Education and Training Needs
Energy Projects Under the NOx Budget Trading Program: Assessment.Center for Labor Research&Education, Donald
A Review of Programs in Indiana,Maryland,Massachusetts, Vial Center on Employment in the Green Economy.
Missouri,New Jersey,New York,and Ohio. http://epa.gov/ www.irle.berkeley.edu/vial.
state loca Id i m ate/d ocu me nts/pdf/ee re_rpt.pdf.
U.S. Environmental Protection Agency.(February 2010).Assessing
the Multiple Benefits of Clean Energy:A Resource for States.
www.epa.gov/statelocalclimate/resources/benefits.html.
December 2012 www.seeaction.energy.gov D-4

This document was developed as a product of the State and Local Energy Efficiency Action Network(SEE Action),facilitated by the U.S.
Department of Energy/U.S. Environmental Protection Agency.Content does not imply an endorsement by the individuals or organizations
that are part of SEE Action working groups,or reflect the views,policies,or otherwise of the federal government.
SEE Action
STATE&LOCAL ENERGY EFFICIENCY ACTION NETWORK