HomeMy WebLinkAbout20230112PAC to Staff 18-3.pdf
Boston | Headquarters
617 492 1400 tel
617 497 7944 fax 800 966 1254 toll free 1000 Winter St Waltham, MA 02451
opiniondynamics.com
PacifiCorp
Idaho Low Income Weatherization Program
Evaluation for Program Years 2013 - 2015
Aaiysha Khursheed, Ph.D.
Principal Consultant
September 11, 2017
opiniondynamics.com Page i
Contributors
Megan Campbell
Vice President
Seth Wayland
Director
Anastacia Bronner
Senior Analyst
opiniondynamics.com Page ii
Table of Contents
1. Executive Summary .............................................................................................................................. 1
2. Introduction ........................................................................................................................................... 7
3. Data Sources ....................................................................................................................................... 10
3.1 Program tracking data .............................................................................................................. 10
3.2 Client consumption data .......................................................................................................... 11
3.3 Monthly external payment and arrearage records ................................................................. 11
3.4 Agency interviews and participant survey data ...................................................................... 12
4. Impact Evaluation ............................................................................................................................... 13
4.1 Methodology ............................................................................................................................. 13
4.2 Results ...................................................................................................................................... 16
5. Process Evaluation .............................................................................................................................. 18
5.1 Agency perspective ................................................................................................................... 18
5.2 Participant perspective ............................................................................................................ 20
6. Payment and Arrearage Analyses for Non-Energy Benefits ............................................................. 28
6.1 Methodology ............................................................................................................................. 28
6.2 Results ...................................................................................................................................... 28
7. Cost-Effectiveness............................................................................................................................... 30
8. Conclusions and Recommendations ................................................................................................. 34
Appendix A: Alternative Model Specifications .......................................................................................... 36
Appendix B: Alternative Financing Documentation ................................................................................. 38
opiniondynamics.com Page iii
Table of Acronyms
Acronyms Meaning
ARRA American Reinvestment and Recovery Act
CAPAI Community Action Partnership Association of Idaho
CSA Conditional Savings Analysis
CFL Compact Fluorescent Light Bulb
EICAP Eastern Idaho Community Action Partnership
IDHW Idaho Department of Health and Welfare
kWh Kilowatt-hour
LIHEAP Low Income Home Energy Assistance Program
LIWP Low Income Weatherization Program
NEB Non-Energy Benefit
PCT Participant Cost Test
PTRC PacifiCorp Total Resource Cost Test
PUC Public Utilities Commission
RIM Ratepayer Impact Measure Test
SEICAA SouthEastern Idaho Community Action Agency
SIR Savings-to-Investment Ratio
TRC Total Resource Cost
UCT Utility Cost Test
USDHHS United States Department of Health & Human Services
USDOE, DOE United States Department of Energy
WAP Weatherization Assistance Program
Executive Summary
opiniondynamics.com Page 1
1. Executive Summary
Opinion Dynamics presents its evaluation findings for the Rocky Mountain Power Low Income Weatherization
Program (referred to as the “Program” throughout this report) in operation in the state of Idaho during the
2013 through 2015 program years. We performed both an impact and process evaluation and results from
these are presented in the report. Additionally, we conducted payment and arrearage analyses to estimate
non-energy program benefits. In this report, we also include cost-effectiveness test results using several
approaches. Navigant Consulting performed the cost-effectiveness tests.
Two Idaho non-profit agencies known for serving low income communities implement the Program:
SouthEastern Idaho Community Action Agency (SEICAA) and Eastern Idaho Community Action Partnership
(EICAP). These agencies provide energy efficiency services mostly targeted towards weatherization to existing
single family (including manufactured) and multi-family homes, so long as the multi-family property is at least
66% occupied by low income qualifying tenants. “Low Income” qualifications follow federal guidelines and
eligibility is based on 200% of federal poverty guidelines. Clients receive energy efficiency measures at no cost
to them. Instead, the Rocky Mountain Power reimburses the agencies for 85% of the installation cost. The
agencies receive additional funds to operate the program from the U.S. Department of Energy (USDOE) and
the U.S. Department of Health and Human Services (USDHHS). These funds are allocated to the Idaho
Department of Health and Welfare (IDHW) and administered on its behalf by the Community Action Partnership
Association of Idaho (CAPAI). CAPAI also provides oversight of the weatherization agencies. Agencies are also
reimbursed for administrative costs.
Opinion Dynamics conducted an evaluation of the Program on behalf of the utility for the 2013 through 2015
program years. The evaluation objectives were to: (1) document and measure effects of the program (energy
and non-energy); and (2) identify areas of potential improvement. To quantify energy benefits, we conducted
an impact evaluation using a consumption analysis with a comparison group to estimate the ex-post net
annual energy savings attributable to the Program. To quantify non-energy benefits such as reduced costs and
external payments, we conducted an assistance payment analysis and an arrearage analysis of the treatment
and comparison groups. We also conducted a process evaluation based on a program materials review, in-
depth interviews with agency staff (SEICAA and EICAP), and client responses to a telephone survey. The
telephone survey asked about client satisfaction with the program and implementers, program barriers and
bottlenecks, best practices, and any opportunities for improvement. Last, this report includes the cost-
effectiveness test results supplied by Navigant Consulting.
1.1.1 Impact Results
For the impact evaluation, we verified Program participation through participant telephone surveys. All
surveyed participants (n=21) verified they participated in the program and received measures. We conducted
a consumption analysis to estimate the electric savings. We applied a Conditional Savings Analysis (CSA)
model to estimate weather-normalized, Program-induced energy (kWh) savings based on differences between
participant consumption data and the comparison group. The result shows that the average annual net energy
savings per participant for the 2013-2015 program years is 1,185 kWh.
This estimate is lower than the energy savings estimated for the Program in the previous evaluation. Lower
savings can result from a variety of factors such as the mix of measures installed, as well as characteristics of
the clients who participated in the Program. During the 2013-2015 program years, no participants replaced
furnaces, but a total of 16 furnaces were replaced during the 2010-2012 program years. Furnace
replacements are a significant source of energy savings, particularly if the previous units are very old. Another
contributing factor to smaller energy savings may be from occupancy changes. Over one-quarter of survey
Executive Summary
opiniondynamics.com Page 2
respondents indicated that someone in the household retired or became unemployed since the measures
were installed which may have increased the hours of use for heating and water heating which could then
decrease energy savings. In Table 1, we present the ex-post net savings for each program year and in total.
Overall, the Program achieved 90% of its ex-ante gross savings for the evaluation period.
Table 1. Ex-Ante Gross and Ex Post Net Energy Savings (kWh)
Program Year Participation
Ex-Ante Gross
Energy Savings
(kWh)
Ex-Post Net
Energy Savings
(kWh)
Realization
Rate
2013 74 101,771 87,690 86%
2014 41 52,320 48,585 93%
2015 53 68,016 62,805 92%
Total 168 222,107 199,080 90%
The net savings may reflect both measure savings and behavior changes given that many participants took
recommended actions to save energy beyond the measures installed. The Program is installing deep energy
savings measures that will likely provide persistent savings over time as many of the measures have a long
effective useful life such as insulation. Further, most participants will reap these savings over a long period
since most of them (81%) own their homes. The Program’s decision to move from CFLs to LEDs in 2016 is a
solid one given the current lighting market conditions, i.e. Energy Independence and Security Act (EISA)
legislation is slowly removing incandescents from store shelves and CFLs are more prevalent in homes. Half
of the survey respondent (52%) said they already had CFLs in their home before participating. Forty percent
(n=7 out of 17) stated that all CFL bulbs were still installed, which means that most program participants
removed some or all of the CFL bulbs. The Program’s decision to move from CFLs to LEDs will likely reduce
the removal rate.
1.1.2 Process Results
The process evaluation examined program operations from multiple perspectives. Rocky Mountain Power and
its implementers, SEICAA and EICAP, have worked together for several years to deliver the Program. Over this
time, they have developed expertise in delivering the program despite its complex funding mechanisms.
Combining the funds from Rocky Mountain Power with additional money from government organizations allows
the program to reach more utility clients and demonstrates a best practice in low income energy efficiency
program delivery.1 It is a common practice for utilities to work with community action agencies to bring their
energy efficiency programs to low income households since these organizations generally have well-
established relationships with them already.
The agencies can serve most clients that qualify relatively quickly; most often within three months of applying
with some exceptions. More than half of the surveyed participants (62%) reported wait times of less than 3
months. Still, approximately 10% of clients stated that they had to wait a year or longer from their application
processing date. SEICAA served its entire waiting list for the Program while EICAP reported that some clients
on its waiting list may not receive services for up to two to three years. This may be indicative of the difference
1 Kushler, Martin, York, Dan and Witte, Patti, “Meeting Essential Needs: The Results of a National Search for Exemplary Utility-Funded
Low-Income Energy Efficiency Programs”, ACEEE Report Number U053, September 2005.
Executive Summary
opiniondynamics.com Page 3
between the agencies in terms of how many clients they serve, as EICAP serves more clients than SEICAA.
Amongst participants, 75% received EICAP services while 25% received SEICAA services. The agencies both
noted that they work to restructure their waiting lists based on federally mandated Program priorities (such as
serving the elderly, disabled, and homes with young children). EICAP noted that it reviews the wait list daily to
re-prioritize applicants based on how long they have been waiting for services, as well as by cost of heating as
a proportion of the household’s income.
From the agency perspective, the program is operating smoothly. However, there are two key issues impacting
participation rates and program administrative costs. The first issue is a structural barrier that is very common
in low income weatherization programs across the country. Sometimes, the Program cannot install energy
efficiency measures because other structural or safety issues in the home need to be addressed first and are
not covered by the Program. The second issue is a client awareness issue where clients have difficulty self-
reporting that they have electric heat, which is an eligibility requirement. Clients may say they have electric
heat and the agencies may spend time arriving at the home and discovering that the client does not have
electric heat and, therefore does not qualify for all weatherization measures.
The Program is helping to educate participants on ways to save energy beyond the direct-install measures.
While energy education is not a formal part of this Program2 and is offered through Rocky Mountain Power’s
Low Income Energy Conservation Education Program, agency staff still speak to Program participants about
ways to save energy in the home. Coupling this informal energy efficiency education with home audits and
measure installation is one way implementation staff can take advantage of their visits to help induce
behavioral changes that may further reduce energy costs. It is also considered a best practice of energy
efficiency programs designed to serve low income clients.3 Almost all survey respondents recall receiving
energy education from the Program and found it very helpful.
The Program is also going beyond energy and cost benefits by improving the health, comfort and aesthetics of
the homes. In the telephone survey, we asked program participants if the air quality, appearance, and comfort
were better, the same, or worse after they participated in the program. Eighty-six percent of respondents
reported an improvement in comfort, 43% in air quality, and 48% in home appearance. No one reported that
these home characteristics were worse since participation.
The Program is meeting client needs very well. Participant experience with the Program was very positive. Four
in five (86%) participants reported that they were “completely satisfied” with the Program and 95% would
recommend the program to others; consistent with previous program evaluation results.4
Rocky Mountain Power tried to increase awareness about its sponsorship of the Program with additional
efforts in 2015. For example, clients now receive letters from Rocky Mountain Power thanking them for their
participation after they receive weatherization services through the Program. However, the agencies are
generally credited for the funding more than Rocky Mountain Power. Only 10% of surveyed clients identified
Rocky Mountain Power as a funding source. It may take time for this information campaign to take effect and
2 Rocky Mountain Power provides $25,000 annually for Low Income Energy Conservation Education.
3 Ibid.
4 Smith & Lehmann Consulting and H. Gil Peach & Associates, Idaho Low-Income Weatherization Program Evaluation Report for
Program Years 2010-2012, Prepared for Rocky Mountain Power. January 26, 2015, page 27.
Executive Summary
opiniondynamics.com Page 4
increase awareness concerning Rocky Mountain Power’s sponsorship of the services provided by EICAP and
SEICAA.
1.1.3 Payment and Arrearage Analyses Results
To estimate some non-energy benefits from the Program, we compared the change in external assistance
payments and arrearages for program participants and a comparison group. Table 2 presents the annual
change in assistance payments annually and overall for the evaluation period. Assistance payments
decreased by an average of over 40% for Program participants while it increased by over 60% for the
comparison group. A net reduction in external payments of $112 is the net benefit of the Program.
Table 2. Payment Assistance Amounts Summary for Participants and Comparison Group
In addition to a reduction in external assistance payments, we examined the change in arrearages. An
arrearage is the unpaid ending monthly balance on a customer’s bill. To estimate this non-energy benefit, we
calculated the change in arrearage payments for Program participants and compared this to the change in
arrearage payments for the comparison group. Table 3 presents the findings from this analysis. The net
difference in arrearage payments is $17 per month, since arrearages decrease for the participant group and
increase for the comparison group. However like the analysis above, the net difference does not represent the
non-energy benefit because neither the participant group or the utility benefit from the increased arrearages
paid to the comparison group. The net Program benefit is the $5 reduction in monthly arrearages paid to the
participants of the Program.
Table 3. Arrearage Summary for Participant and Comparison Groups
1.1.4 Cost-Effectiveness Results
Navigant completed cost-effectiveness tests of the Program using various approaches: the PacifiCorp Total
Resource Cost (PTRC) test, Total Resource Cost (TRC) test, Utility Cost (UTC) test, Ratepayer Impact Measure
(RIM) test, and the Participant Cost Test (PCT). Opinion Dynamics and PacifiCorp provided the inputs to
Navigant for their calculations. The PCT was considered “not applicable” and benefit/cost ratios were not
calculated using this approach. The annual and evaluation period benefit/cost ratios are presented in Table
Net
Difference
Pre Post Change %
Change Pre Post Change %
Change Amount
2013 $ 229 $ 128 $ (101)-44% $ 1,464 $ 1,460 $ (4)0% $ 97 $ 101
2014 $ 278 $ 128 $ (149)-54% $ 1,433 $ 2,354 $ 921 64% $ 1,071 $ 149
2015 $ 275 $ 189 $ (86)-31% $ 2,245 $ 4,976 $ 2,731 122% $ 2,817 $ 86
Total $ 260 $ 148 $ (112)-43% $1,714 $2,930 $1,216 62% $ 1,328 $ 112
Year
Participant Group Comparison Group Net
Program
Benefit
Net
Difference
Pre Post Change %
Change Pre Post Change %
Change Amount
Monthly Arrearage $ 38 $ 33 $ (5)-14% $ 28 $ 40 $ 12 43% $ 17 $ 5
Participant Group Arrearage Comparison Group Arrearage Net
Program
Benefit
Executive Summary
opiniondynamics.com Page 5
4 and show that the Low Income Weatherization Program is considered cost-effective based on the PTRC and
TRC tests. Note that this Program uses the PTRC to determine cost-effectiveness.
Table 4. Benefit/Cost Ratios - Low Income Weatherization
1.1.5 Recommendations
Based on the evaluation results, we recommend the following:
Rocky Mountain Power is adhering to best practices by delivering the program through community-
based agencies. SEICAA and EICAP have served as Program implementers on behalf of Rocky
Mountain Power for years. It is a common practice for utilities to work with community action agencies
to bring their energy efficiency programs to low income households since these organizations generally
have well-established relationships with them already. Additionally, these agencies are knowledgeable
about using funding from utilities in combination with government funding to expand the reach of
programs. SEICAA and EICAP both demonstrate their understanding of program processes,
requirements and funding mechanisms. Leveraging these type of agencies is a best practice in low
income weatherization programs. Rocky Mountain Power should continue to use the same Program
implementers moving forward.
Rocky Mountain Power has tried to increase awareness about its funding of the program, given that
the utility provides 85% of the costs of measures installed in participants’ homes. Most participants
cannot recall who funds the Program and those that do often associate it with the agencies instead of
the utility. Only 10% of surveyed clients identified Rocky Mountain Power as the funding source. In
2015, Rocky Mountain Power started to send letters and magnets to participants to thank clients for
participating and to increase awareness of the utilities’ role in the program. These efforts may help
increase association of the Program with Rocky Mountain Power over time but the Program may also
consider branding the agency staff who conduct the audits and installation services by wearing shirts
with the Rocky Mountain Power name and logo.
Long waiting lists to receive weatherization services continue from one agency’s perspective, although
that agency could not decipher the Rocky Mountain Power waiting list versus other utilities. It may not
be a huge issue for Rocky Mountain Power clients given that 62% of survey respondents said the
Program served them within 3 months of applying. SEICAA noted that it served all Rocky Mountain
Power clients that qualified and still had remaining funds. The demand for services may be higher than
what Rocky Mountain Power can provide, particularly for EICAP. However, since EICAP exhausted their
Program funding and SEICCA did not use all of its funding, Rocky Mountain Power may revisit the
funding levels to each agency and consider giving more to EICAP and less to SEICCA.
Though the Program has been well received, it has had declining participation since 2012. The decline
in participation could be due to several factors, including market penetration amongst the eligible
Program Year PTRC TRC UCT RIM PCT
2013 1.23 1.17 0.63 0.4 n/a
2014 1.24 1.18 0.64 0.4 n/a
2015 1.22 1.17 0.63 0.4 n/a
2013-2015 1.23 1.17 0.63 0.4 n/a
Executive Summary
opiniondynamics.com Page 6
population and depletion of American Recovery and Reinvestment Act (ARRA) funding. We recommend
that Rocky Mountain Power take a historical look at participation amongst its low income population
that likely has electric heat to determine how much of the market has been penetrated thus far. This
exercise could also help to identify and target households that have not participated yet.
The Program could reduce costs if agencies can verify that a client has electric heat before visiting the
home. Clients have difficulty with correctly identifying whether their home uses electric heating.
Currently, the agencies rely on clients to tell them if they have electric heat and then verify it by visiting
the home. We recommend that Rocky Mountain Power help the agencies determine if a client has
electric heat through consumption records before visiting the home. The average electric consumption
for low income households with electric heat could help agencies determine if a client is in the general
ballpark before visiting the home.
Finally, the Program is struggling with an issue commonly found in low income weatherization
programs throughout the country, i.e., overcoming the structural barriers to installing weatherization
measures. These structural barriers are an issue impeding participation and cost-effectiveness. This
issue is a quandary to most utilities who need to allocate funds directly to energy saving improvements,
for cost-effectiveness standards, instead of structural and safety improvements that do not directly
lead to energy savings. While other funding sources can help, it often is not enough. For most utilities,
this remains an unsolvable dilemma. However, one electric cooperative in Arkansas advocated for a
new tariff in the state that allowed for an innovative financing solution that directly solved this issue.
The Pay-As-You-Save model, allows the utility to fund both structural and energy improvements and
provides immediate net savings for the client. The client does not incur a debt obligation while the
utility benefits from a low risk path to cost recovery through a charge on the bill that is less than the
estimated savings from the upgrades. We recommend that Rocky Mountain Power staff explore this
innovating financing tariff that allowed a utility to address both structural and energy improvements
through its low income weatherization program at no cost to the client. More information on this
innovate tariff and how it operates can be found in the embedded documents in Appendix B.
Introduction
opiniondynamics.com Page 7
2. Introduction
Rocky Mountain Power’s Low Income Weatherization Program (the “Program”) provides energy efficiency
services to eligible residential clients through a partnership with two non-profit weatherization agencies in
Idaho: Eastern Idaho Community Action Partnership (EICAP)5 and SouthEastern Idaho Community Action
Agency (SEICCA).6 Partnering with agencies that historically serve Idaho’s low income communities provides
Rocky Mountain Power with access to the clients targeted by this program.
Rocky Mountain Power funds 85% of the cost of approved measures received by participants. To fund the
remainder, the agencies leverage government funding through the Idaho Department of Health and Welfare
(IDHW). The original sources of these funds come from the United States Department of Energy (USDOE) and
the United States Department of Health and Human Services (USDHHS). These funds are administered by the
Community Action Partnership Association of Idaho (CAPAI) and directed to SEICAA and EICAP. Leveraging
utility, state and federal funding sources allows these agencies to provide comprehensive weatherization
services to more low income households than they may have otherwise. Other exemplary utility-funded low
income energy efficiency programs also bring together multiple funding sources and implement programs
through social service agencies. We show the sources of funding and roles of oversight and implementation
in Figure 1.
Figure 1. Funding and Oversight for Rocky Mountain Power’s Low Income Weatherization Program
5 EICAP serves Bonneville County, Butte County, Clark County, Fremont County, Jefferson County, Lemhi County, Madison County and
Teton County
6 SEICAA serves Bannock County, Bear Lake County, Bingham County, Caribou County, Franklin County, Oneida County and Power
County
Introduction
opiniondynamics.com Page 8
2.1.1 Program Implementation
Program implementation by SEICAA and EICAP involves the following steps, which are described in further
detail in the 2015 Idaho Energy Efficiency and Peak Reduction Annual Report:
income verification based on CAPAI guidelines to ensure that participants qualify for program
participation,
energy audit using a U.S. Department of Energy approved tool to determine measures that are cost
effective to install,
installation of measures that have a Savings Investment Ratio of 1.0 or greater,
post-inspections of all projects, and
billing notification to Rocky Mountain Power, which includes the measures installed and the associated
cost of each project, along with the associated invoice.
The Program is available to all existing single family and multi-family residential units, so long as the multi-
family property is at least 66% occupied by low income qualifying tenants. “Low income” qualifications follow
Federal low-income guidelines and income eligibility is based on 200% of federal poverty guidelines.
Agencies directly install measures for clients based on heating fuel-type and need. Measures vary by
household, are classified as either “major” or “supplemental”, and could include the following during the
evaluation period: CFLs, water pipe insulation, showerheads, aerators, infiltration, replacement windows,
thermal doors, thermostats, health and safety measures, electric furnace repair and replacement, ceiling,
floor, wall, and duct, insulation, attic ventilation, water heater repair and replacement and refrigerators.
2.1.2 Evaluation Objectives
Below we list the objectives of our evaluation of the Rocky Mountain Power Low Income Weatherization
Program in Idaho and we include in parentheses the evaluation type in which the objective is covered:
Document and measure effects of the Program (impact and process)
Verify measure installation and savings (impact)
Review Program operations (process)
Document all other funding used by agencies to provide no-charge services to participants (process)
Quantify non-energy benefits through payment analysis (payment/arrearage analysis)
Provide data to support Program cost-effectiveness assessments (impact and payment/arrearage
analyses)
Identify areas of potential improvement (impact and process)
Document compliance with regulatory requirements (process)
Survey participants and agency staff (process)
Introduction
opiniondynamics.com Page 9
In the remainder of the report, we include a description of the data collection and methodologies used to
conduct the study, a presentation of the impact evaluation, the findings from the process evaluation, the
assistance payment and arrearage analyses, and cost-effectiveness results.
Data Sources
opiniondynamics.com Page 10
3. Data Sources
In this section, we present the data sources used in this evaluation.
3.1 Program tracking data
We requested and received program tracking data for program years 2013 through 2016 to support both
impact and process evaluation. These data are tracked at the measure level therefore program participants
who received more than one measure or treatment are listed multiple times. Our examination of the data
revealed that Rocky Mountain Power Company changed their Program tracking system after 2013, therefore
some of the variables provided in the 2014-2016 program tracking data were not provided in the 2013 data.
However, we received all necessary data fields to conduct both the impact and process evaluation components
of the study.
We received the following key variables in the 2013 program tracking data:
Client name
Project name
Project ID
Cost recovery date
Measure installed
kWh/year savings
Direct install costs
Measure costs
Account number (client identifier, provided in a different data extract)
The Program tracking data system used for 2014 participants and beyond differed from the system used in
2013. We received more variables per record, which was at the measure level. We received the following key
variables in the 2014-2016 program tracking data:
Client name, address, and phone number
Project name
Project ID
Cost recovery date
Project creation date
Project last update date
Measure category, type, sub-type, and name
Data Sources
opiniondynamics.com Page 11
Direct install costs
Measure costs
Bill account number (a client identifier and is the same as Account number in 2013 program tracking
data)
Primary utility number (client identifier)
The Program tracking data systems did not include kWh/year savings at the measure level and assumed the
same average savings per home. Because we conducted a consumption analysis for the impact evaluation,
the kWh/year savings at the measure or participant level were not needed.
Note that while we did not evaluate the 2016 program year, we requested these data for the consumption
analysis as well as the payment analysis. We used future program participants as a comparison group where
participants of the program were matched to them based on zip code and average daily consumption.
We used the program tracking data to identify program participants and the measures they had installed to
develop the participant telephone survey sample. During the survey, we asked respondents to verify their
participation.
3.2 Client consumption data
We received client consumption data from January 2012 through November 2016 for clients who participated
in the Program during the 2013 through 2016 program years. The 2012 consumption data allowed us to
establish baseline energy usage for those clients who participated in the Program during the 2013 through
2015 evaluation years and for the comparison group. These data included monthly kWh usage and one of a
few different client identifiers (e.g., bill account number or a primary utility number) thereby allowing us to
relate the consumption data to Program tracking data.
3.3 Monthly external payment and arrearage records
The payment and arrearage analyses relied on monthly client assistance payments received and monthly
arrearages amongst participants and the comparison group. Key client payment data we received included
the following variables for program participants:
Client identifier
Date of billed amount (generally billed monthly)
Balance forward amount (represents monthly customer arrearages)
Client assistance payment amount
Client assistance payment date
Data Sources
opiniondynamics.com Page 12
3.4 Agency interviews and participant survey data
Primary data collection activities included in-depth interviews with staff members at the SouthEastern Idaho
Community Action Agency (SEICAA) and Eastern Idaho Community Action Partnership (EICAP). We also
conducted a participant telephone survey. The agency interviews helped inform our review of Program
operations, compliance with regulatory requirements, as well as major accomplishments and challenges
related to Program implementation. We used information gathered through the participant telephone survey
to verify the installation of measures, estimate lighting in-service rates, and inform process related Program
findings.
Impact Evaluation
opiniondynamics.com Page 13
4. Impact Evaluation
A total of 168 clients participated in the program over the 2013 through 2015 years. In the participant
telephone survey, we asked respondents whether they recall someone coming to their home to provide
weatherization services and perform energy efficiency upgrades. All surveyed respondents (n=21) confirmed
their participation.7 A list of the various measures installed from the most common, compact fluorescent light
bulbs, to the least common, water heater replacement, is presented in Table 5 below. Other common
measures include water pipe insulation, infiltration, windows, and thermal doors.
Table 5. Idaho Participation Counts and Measures for Program Years 2013 to 2015
Measures 2013 2014 2015 Total Percent
Treated
Total # of Treated Homes 74 41 53 168 100%
Compact Fluorescent Light Bulbs 68 40 49 157 93%
Water Pipe Insulation 58 35 49 142 85%
Infiltration 47 37 44 128 76%
Replacement Windows 38 23 39 100 60%
Thermal Doors 35 28 33 96 57%
Furnace Repair 34 26 29 89 53%
Health & Safety Measures 26 24 30 80 48%
Ceiling Insulation 26 24 27 77 46%
Floor Insulation 20 15 21 56 33%
Attic Ventilation 23 11 21 55 33%
Duct Insulation 19 12 10 41 24%
Water Heater Repair 6 4 9 19 11%
Wall Insulation 6 4 5 15 9%
Refrigerator Replacement 3 3 8 14 8%
Water Heater Replacement - 1 1 2 1%
4.1 Methodology
We conducted a consumption analysis to estimate the electric energy savings. Our methodology compares
pre- and post-participation energy usage, using future participants as a comparison group. This is called a
Variation-in-Adoption method, and it is one of the recommended methods to use when it is not possible to do
a randomized control test.8 Since this is a three-year study, pre-participation usage for 2014 and 2015
7 Participant telephone survey sample only included participants from 2014 and 2015 to help mitigate recall bias.
8 SEE Action, “Evaluation, Measurement, and Verification (EM&V) of Residential Behavior-Based Energy Efficiency Programs: Issues
and Recommendations”, DOE/EE-0734, May 2012, p. 17.
Impact Evaluation
opiniondynamics.com Page 14
participants serves as a comparison for 2013 participants. Likewise, pre-participation usage for 2015
participants serves as a comparison for 2014 participants. To get a comparison for 2015 participants, we
include pre-participation usage for 2016 participants in the model.
We used comparison group matching to ensure that our comparison group was as similar as possible to
participants. For each participant in 2013-2015, we compared their pre-participation monthly bills to the
corresponding monthly bills for each possible comparison group match (using only pre-participation data for
the control group client, also). We then took the difference in kWh usage for each matched monthly pair and
squared it. We developed a score equal to the sum of squared differences across all available months of pre-
participation data for each possible participant-comparison group match. Pairs with the lowest scores indicate
the best comparison group match for each participant based on similar electric usage patterns and levels. We
used these scores, in combination with other geographic data, to build and test different comparison group
specifications within the modeling process.
After selecting the comparison group, we built a Conditional Savings Analysis (CSA) model to estimate weather-
normalized, program-induced energy (kWh) savings based on differences in participant and comparison group
data. We identified program-induced energy savings by combining participant tracking data with client
consumption data to classify pre- and post-participation periods for each individual participant based on the
month their measures were installed.
Next, we weather normalized the model by including variables that account for changing weather conditions
from year to year. We used zip codes for each participant to locate the nearest National Oceanic and
Atmospheric Administration (NOAA) weather station with consistently valid hourly data and identified five valid
stations for Idaho clients.9 We next converted the hourly data into the monthly Heating Degree Day10 and
Cooling Degree Day11 data needed for analysis of monthly consumption. Last, we included a monthly index in
the model to provide information on time trends that appear across all clients, both participants and
comparison clients.
To automatically account for all unknowns that vary by client (such as square footage, etc.), we used the
9 The nearest NOAA weather station with reliable hourly data was found without paying attention to what state the weather station was
located in. That means the nearest station for an Idaho client was not necessarily in Idaho. There were five weather stations matched
to Idaho clients in this study:
Driggs–Reed Memorial Airport, Driggs, ID 83422
Idaho Falls Regional Airport, Idaho Falls, ID 83402
Pocatello Regional Airport, Pocatello, ID 83204
Logan-Cache Airport, Logan, UT 84321
Rexburg–Madison County Airport, Rexburg, ID 83440
For occasional occurrences of missing hourly data within a weather station series, we replaced the missing data with an average of
temperatures from the other weather stations with reliable data. The data from the other stations is weighted based on 1/squared
distance between the two stations. Consequently, a station twice as far away receives ¼ of the weight in the calculation of the average.
10 Heating Degree Day = 65 – Daily Average Temperature; if HDD < 0 then HDD = 0. The HDD is calculated for each day, then summed
over the month to get monthly HDD.
11 Cooling Degree Day = Daily Average Temperature – 65; if CDD < 0 then CDD = 0. The CDD is calculated for each day, then summed
over the month to get monthly CDD.
Impact Evaluation
opiniondynamics.com Page 15
following fixed-effects regression model specification:
𝐴𝐷𝐶𝑘𝑡=𝑎𝑘
+𝑎1𝑀𝑜𝑛𝑡ℎ𝑡
+𝑎2𝐻𝑑𝑑𝐷𝑡
+𝑎3𝐶𝑑𝑑𝐷𝑡
+𝑎4𝑃𝑜𝑠𝑡𝑘𝑡
Where:
𝐴𝐷𝐶𝑘𝑡 = Average Daily kWh Consumption of client k during month t
𝑎𝑘 = Fixed effect of client k
𝑀𝑜𝑛𝑡ℎ𝑡 = Number of months since January 2012 for month t
𝐻𝑑𝑑𝐷𝑡 = Average Heating Degree Days per day during month t
𝐶𝑑𝑑𝐷𝑡 = Average Cooling Degree Days per day during month t
𝑃𝑜𝑠𝑡𝑘𝑡 = A 0/1 binary variable equal to 1 for client k in month t if their LIW
measures have already been installed
4.1.1 Description of the Data
To begin our consumption analysis, we first prepared the data by matching Program participants to the
available billing records. We did so as we felt it important to include billing records only if the same client was
in the same premise for a sufficient amount of time during the study period. This is because many of the
measures create savings related to space heating use, which can vary significantly depending on the comfort
level preferred by the occupant. For example, if measures are installed in a home and a new occupant moves
in shortly after who likes to keep their home warmer, measurement of the true energy savings from the
measures would be obscured by behavior changes. Consequently, our consumption analysis only includes
monthly billing records for clients who resided at the same premise for at least 11 months before and 11
months after the measures were installed. Due to the seasonal nature of savings related to space heat and
cooling, we recognize the importance of including as much of a full year of data as possible for reporting
average annual savings. These requirements left 135 participants in the analysis dataset, which is equal to
approximately 80% of all clients who participated in 2013-2015. They are spread across participation years
as shown in Table 6.
Table 6. Participants with Valid Data for Consumption Analysis
Year Measures
Installed
Number of
Participants
2012 57
2013 35
2014 43
2015 53
Total 135
After identifying program participants with sufficient valid consumption data, we next identified the best
matched comparison client for each participant. Selecting the top three comparison group matches for each
participant using lowest match scores is a good balance between getting a tight match and compensating for
cases with a low number of pre-participation month matches. Note that the same comparison group client is
often in the top three matches for more than one participant. Regardless of the number of matches, each
Impact Evaluation
opiniondynamics.com Page 16
comparison group client is included in the model dataset only once.
Using the top three matches algorithm, we found 405 matches for the 135 participants. There are 150 unique
clients within the group of 405 top three matches. Twenty of these comparison group clients are from the 2016
participant group. Consumption data used for analysis covers 2012 through 2016, to include both pre-
participation data for 2013 participants and post-period comparison data for 2015 participants.
4.2 Results
We produced the results presented in Table 7 when we ran the model with 135 participants and the matched
comparison group from the top three matches algorithm.
Table 7. Results of the Consumption Analysis Model using Top Three Matched Control Group
Variable DF Parameter
Estimate
Standard
Error t Value Pr > |t|
Intercept 1 -0.0027 0.16104 -0.02 0.9866
Month 1 -0.04512 0.01631 -2.77 0.0057
HddD 1 1.35511 0.01344 100.85 <.0001
CddD 1 1.54213 0.1028 15 <.0001
Post 1 -3.24594 0.56812 -5.71 <.0001
As the parameter estimate on the Post variable indicates, we find an average savings of 3.25 kWh per day
after Program measures are installed. This translates to 1,185 kWh of savings per year on a weather-
normalized annual basis. All coefficients are statistically significant at the 95% confidence level or better and
the adjusted R-squared for the model is 0.634.
We built alternative models to test the consistency of the savings estimate from the basic model. Based on
the similarities in energy savings estimates across the model specifications, we feel confident in our annual
per participant savings estimate of 1,185 kWh per year. Results from these models are in Appendix A.
4.2.1 Ex Post Net Energy Savings from the Program
As shown, the average annual net energy savings per participant for the 2013-2015 program years is
estimated as 1,185 kWh. In Table 8, we present the annual ex-ante gross and ex-post net energy savings for
the Program.12 The net savings realization rate is 90% for the 2013-2015 evaluation period.
12 We retrieved ex-ante gross energy savings by year from Rocky Mountain Power’s Idaho Energy Efficiency and Peak Reduction Annual
Reports for the years 2013 through 2015.
Impact Evaluation
opiniondynamics.com Page 17
Table 8. Ex-Ante Gross and Ex Post Net Energy Savings (kWh)
Program Year Participation
Ex-Ante Gross
Energy Savings
(kWh)
Ex-Post Net Energy
Savings (kWh)13
Realization
Rate
2013 74 101,771 87,690 86%
2014 41 52,320 48,585 93%
2015 53 68,016 62,805 92%
Total 168 222,107 199,080 90%
4.2.2 Comparison to Previous Year’s Savings Estimate
The net savings estimate per participant, 1,185 kWh, is approximately 55% of the previous evaluation period
(2010 through 2012). Lower savings can result from a variety of factors such as the mix of measures installed,
as well as characteristics of the clients who participated in the Program. Program tracking data shows that no
furnaces were replaced during the 2013-2015 program years, but a total of 16 furnaces were replaced during
the 2010-2012 program years. Another contributing factor is occupancy changes. Over one-quarter of survey
respondents indicated that someone in the household retired or became unemployed since the measures
were installed which may have increased the hours of use for heating and water heating which could then
decrease energy savings.
4.2.3 CFL Persistence
To get a sense of the persistence of CFLs installed through the Program, we inquired whether participants still
had the bulbs installed. Forty percent (n=7 out of 17) stated that all of the CFL bulbs were still installed, which
means that most program participants removed some or all of the CFL bulbs. Those who replaced bulbs noted
a mix of bulb types used including incandescents and LEDs. The Program’s decision to move from CFLs to
LEDs will likely reduce the removal rate.
13 The annual ex post net energy savings estimate of 1,185 kWh per participant is multiplied by the number of participants to arrive at
the yearly ex post net energy savings in the table.
Process Evaluation
opiniondynamics.com Page 18
5. Process Evaluation
Notably, the Program’s popularity has been declining since 2012 (see Figure 2). It is uncertain if the number
of participants has reduced because it has become more difficult to serve clients in a timely manner, because
ARRA funding is no longer available to help support weatherization efforts, or because fewer clients are signing
up to participate in the program. Regardless, the number of participants served by the program during this
evaluation period is far smaller than it has been in previous years. In this process evaluation, we examined
the Program’s operations from the perspective of the agencies and participants.
Figure 2. Number of LIWP Participants from 2007 - 2015
5.1 Agency perspective
We conducted a total of two agency interviews in December 2016. One was with a representative from EICAP
and the other included two staff members from SEICCA. These interviews were conducted to gain a deeper
understanding of the Program’s operations and any key areas of improvement. We present each agency’s
perspective in the subsections below. Notably, 75% of Program participants received EICAP Program services
and 25% received SEICCA services.
5.1.1 Eastern Idaho Community Action Partnership (EICAP)
EICAP serves a larger number of Rocky Mountain Power clients and successfully used all of its available
Program funds. EICAP has additional state and federal funding sources available to implement weatherization
services. Since Rocky Mountain Power covers 85% of program implementation costs, EICAP mostly uses
USDOE funding to make up the remaining 15%.
0
20
40
60
80
100
120
2007 2008 2009 2010 2011 2012 2013 2014 2015
Nu
m
b
e
r
o
f
P
a
r
t
i
c
i
p
a
n
t
s
Year
Process Evaluation
opiniondynamics.com Page 19
To receive weatherization services from EICAP, residents complete an application, which is then reviewed by
the agency. If an applicant is eligible, he or she is put on a waiting list. EICAP prioritizes households with young
children or with elderly or disabled residents. The agency reviews the waiting list daily to re-prioritize applicants
based on how long they have been waiting for services, as well as the ratio of heating cost to household
income. It can take up to three years to receive weatherization services, though this wait list is not specific to
Rocky Mountain Power applicants since EICAP implements weatherization programs for other agencies as
well.
Once an applicant comes up on the waiting list, EICAP sends out a letter and waits to hear back from the
applicant. If no one responds, EICAP sends out a second letter. The weatherization director noted that during
the fall and winter, applicants tend to be quick to respond and engage in the process. This is reasonable given
clients must feel the effect of the cold strongest during these times. The response back from applicants is not
as strong in the spring and summer since they do not feel the immediate need for weatherization. Those who
do not respond are moved to a second waiting list and will be contacted again later.
The EICAP staff was asked about barriers to participation and challenges operating the Program but did not
think there were any saying the program is a “win-win situation”.
5.1.2 SouthEastern Idaho Community Action Agency (SEICAA)
Funding for SEICCA low income weatherization services comes from a variety of state and federal sources
such as USDOE, USDHHS, LIHEAP, and IDHW, in addition to Rocky Mountain Power. Starting in 2015, the
agency keeps records of the funding sources by program participant. Rocky Mountain Power funding seems
to be sufficient to meet demand as SEICAA did not use all available Rocky Mountain Power funds. As such,
SEICCA does not typically have an issue immediately serving Rocky Mountain Power clients who qualify. Clients
who call from other utility service territories are put on a waiting list, which was estimated to include between
200 and 400 names in their seven-county service area. Though they have an extensive waiting list of clients
from other service territories, SEICAA prioritizes households with young children, elderly or disabled residents,
or homes without working heat or a working water heater.
Agency staff indicated that the Program is running smoothly from their perspective but noted the following
challenges:
One key challenge in operating weatherization program sponsored by several different funding
sources, is that the agency must keep track of the variances by program. The programs do not offer
the same measures and have different eligibility requirements.
Clients sometimes are unsure of whether their heating source is electric. They may think they have
electric heat, but when SEICAA visits they home they discover it does not qualify for the Program
because it has non-electric heat. Sending out auditors to homes that are not eligible for the program
leads to increased operating costs without commensurate benefits from energy savings through
weatherization.
Safety and structural issues in the home are barriers to program participation and contribute to
program costs without energy saving benefits. If an auditor comes to a home and finds faulty wiring,
excessive mold, lead paint, or sewer leaks that could be harmful to the health of crews who would
weatherize the home, clients are asked to deal with these concerns before Program measures can be
installed. Residents may not have the funds to address these issues or they may rent their homes from
a homeowner who chooses not to address these issues. SEICAA staff said it would be nice if they could
use program funds for roof repairs and sewer leaks. They do have access to crisis funding for plumbing
Process Evaluation
opiniondynamics.com Page 20
and minor leaks, but there is not enough funding available to cover large scale roof repairs and sewer
leaks. As noted in Rocky Mountain Power’s Electric Service Schedule 21 which addresses the Low
Income Weatherization Program, “reimbursements related to health and safety measures are limited
to 15% of the annual cost of total jobs performed by the agency.” Some funding is therefore available,
but not enough to cover large scale issues.
5.2 Participant perspective
The evaluation team attempted to reach a census of clients who participated in the Program in 2014 and
2015 with a telephone survey. Participants from 2013 were not included to avoid recall bias, given the amount
of time that has passed since these participants received weatherization services through the Program. Of the
94 clients who participated in 2014-2015, we had valid phone numbers for 91. A total of 21 participants
completed telephone interviews, yielding a response rate of 33% and cooperation rate of 75%.14 (see Table
9).
Table 9. Idaho Client Telephone Survey
Population Frame Unique Telephone
Numbers
Final Survey
Responses
Survey Response
Rate
Survey
Cooperation Rate
94 91 21 33% 75%
The call center attempted to reach participants multiple times. Table 10 lists the survey disposition categories.
14 Response rate is calculated using American Association for Public Opinion Research (AAPOR) Response Rate 3.
Process Evaluation
opiniondynamics.com Page 21
Table 10. Participant Survey Disposition
Survey Disposition Sample
Completed 21
Disconnected phone 22
Not available callback 12
Answering machine 11
No answer 4
Not available 4
Hard Refusal - Do not call 4
Initial refusal 3
Client said wrong number 3
Language problems 2
Non-specific callback/secretary 2
Busy 1
Business/Residential phone 1
Computer tone 1
Total 91
We used this survey to collect data about participant household characteristics and Program experience.
Based on demographic data provided by clients during the participant survey, approximately 62% (n=13)
stated that they reside in single family or manufactured homes and one-third reported living in mobile homes
(n=7). A total of 81% (n=17) own their homes with the remaining 19% renting their residences. Ninety percent
of surveyed participants also self-reported that their homes were built before 1996.
5.2.1 Program Awareness
Participants were asked how they heard about the Program. Figure 3 shows that most participants heard about
the program by word of mouth from family, friends, and neighbors (43%). Fourteen percent of participants
learned about it through marketing through television, newspapers, and/or flyers.
Figure 3. How Participants Learned of the Program (n=21)
24%
5%
14%
14%
43%
0%10%20%30%40%50%
Don't know
Through another health assistance program
Through another energy assistance program
Advertisement (TV/newspaper/flyer)
Word of mouth
Process Evaluation
opiniondynamics.com Page 22
Most participants are not able to identify the funding source for the Program. As seen in Figure 4, participants
who could identify a funding source often associated the Program with the agency not Rocky Mountain Power.
The agency staff from SEICAA reported that implementation staff places a sign in the front yards of homes to
acknowledge both SEICAA and Rocky Mountain Power are providing the weatherization services.
Figure 4. Participant Awareness of Program Funding Sources (n=21)
Most surveyed participants (62%) reported receiving weatherization services within three months of
submitting their application.
Figure 5. Time between Application Process to Receiving Weatherization Services (n=21)
5.2.2 Energy Education
The Program does not offer energy education formally, however, Figure 6 shows 90% of survey respondents
learned about ways to save energy from the agency staff, and many of them (78%, n=19) took some
recommended energy saving actions (Figure 7). Even though the Program does not officially include energy
education, the opportunity to present energy saving recommendations during audits or measure installations
has had a positive impact on program participants.
67%5%
10%
19%
Don't Know State Funds Rocky Mountain Power Agency
10%
14%
10%
5%
33%
29%
0%20%40%60%80%100%
Don't know
More than a year
Six months to a year
Three to six months
One to three months
Less than one month
Process Evaluation
opiniondynamics.com Page 23
Figure 6. Weatherization Staff Provided information on Ways to Save Energy in the Home (n=21)
Figure 7. Participants Who Took Energy Saving Actions (n=19)
Participants provided positive feedback on the energy education received informally, as most participants
indicated the energy education they received was “extremely helpful” (Figure 8).
90%
10%
Yes Don'tKnow
79%16%5%
0%10%20%30%40%50%60%70%80%90%100%
Yes No Don’t Know
Process Evaluation
opiniondynamics.com Page 24
Figure 8. Helpfulness of Energy Education (n=21)
Scale from 0 to 10 where 0 is “Not at All Helpful” and 10 is “Extremely Helpful”
In addition to ways to save energy in the house, 81% of participants indicated the weatherization staff
discussed ways to improve health and safety in the home (Figure 9).
Figure 9. Ways to Improve Health and Safety in the Home (n=21)
5.2.3 Program Delivery and Satisfaction
Participant feedback was highly positive as well. Most participants were “completely satisfied” with the
Program, as seen in Figure 10. Further, 95% of participants said they would recommend it to others (Figure
11).
21%74%5%
0%20%40%60%80%100%
5-7 Rating 8-10 Rating Don't Know
Mean
8.7
81%
10%
10%
Yes No Don’t Know
Process Evaluation
opiniondynamics.com Page 25
Figure 10. Program Satisfaction (n=21)
Scale from 0 to 10 where 0 is “Completely Dissatisfied” and 10 is “Completely Satisfied”
Figure 11. Recommend Program to Family and Friends (n=21)
Reflecting high Program satisfaction, a little over half of respondents had no suggestions for improving the
Program. Amongst those who provided suggestions, participants most often requested more repairs in the
home, better work quality or a quicker participation process. The table below includes the verbatim
suggestions from survey respondents.
Participant Recommendations for Program Improvements
Follow through with work faster.
Speeding up approval process.
One fan leaked a little bit, several months after. Better understanding of what they did, looking back it’s a very basic
understanding. We weren’t completely sure on what they were doing, and would be nice to be a little more informed.
Have better quality of materials to work with.
If they could get more money for the program, so they can provide more help for things such as bad heaters.
Weren't allowed to go under the motor home would like to have seen more repairs done to the bottom of the motor
home.
Should have done all the windows.
14%86%
0%20%40%60%80%100%
5-7 Rating 8-10 Rating
95%
5%
Yes No
Mean
8.6
Process Evaluation
opiniondynamics.com Page 26
Participants were pleased with the application process, with 75% stating the process was “Extremely Easy”.
Further, all participants were very pleased with the weatherization staff, all stating “Yes” when asked if the
agency staff was courteous and respectful towards participants and their family members and 86% agreed
the work crew worked carefully to protect the home.
5.2.4 Impact of Program
There were seventeen participants who recalled the weatherization staff installing CFL bulbs. Of those, 59%
(n=10 out of 17) were more satisfied with the CFLs than their previous lighting and 35% stated the lighting
quality was about the same (Figure 12). Understanding the lighting landscape in Idaho amongst low income
clients helps to determine whether they have ceased purchases of incandescents due to EISA legislation and
started to migrate to CFLs without the Program. If so, this would be an argument to stop providing CFLs or to
switch to LEDs instead. Many participants reported they had CFLs prior to receiving the free bulbs (52%). Given
this, the Program made a sound decision to switch from CFLs to LEDs in 2016.
Figure 12. Satisfaction with CFLs (n=17)
As seen in Figure 13, 67% of participants noticed a change in their electric bill and of those 52% (n=11 out
of 14) said their bill was lower following the weatherization services.
6%
35%
59%
0%20%40%60%80%100%
Less satisfied
About the same
More satisfied
Process Evaluation
opiniondynamics.com Page 27
Figure 13. Change Noticed in Electric Bill (n=21)
We also explore non-energy impacts. In the telephone survey, we asked Program participants if the air quality,
appearance, and comfort were better, the same, or worse after they participated. As Figure 14 shows, comfort
of the home improved the most, with 86% noting an improvement. Home appearance and air quality in the
home were better for 43% and 48% of participants as well. This provides further evidence of the positive
impact of the Program beyond energy saving benefits.
Figure 14. Impact of Measures on Home Characteristics (n=21)
67%
33%
Yes No
86%
43%
48%
14%
57%
48%5%
0%20%40%60%80%100%
Comfort
Appearance
Air Quality
Would you say the ___________ of your home is better, the same,
or worse?
Better Same Don't know
Payment and Arrearage Analyses for Non-Energy Benefits
opiniondynamics.com Page 28
6. Payment and Arrearage Analyses for Non-Energy Benefits
We completed payment and arrearage analyses to quantify some non-energy impacts of the Program. We
compared changes to external assistance payments and customer arrearages between Program participants
and a comparison group over the evaluation period. These cost savings serve as non-energy benefit inputs to
calculate cost-effectiveness for the Program.
6.1 Methodology
In addition to the external payment data described in the Data Sources section (Section 3), additional data
used in the analysis came from the Program tracking data. We merged the cost recovery date, which allowed
us to determine the pre- and post- periods based on when the client received the energy efficient measures.15
With these data, we calculated the difference external payments and customer arrearages made during pre-
and post-periods between Program participants. We define the pre-period as the year prior to the cost recovery
date and the post-period as the year after the cost recovery date. For the comparison group, we estimated the
average cost recovery date for all participants and used it for every household in the comparison group.
Opinion Dynamics first reviewed the participant and comparison group external payment and arrearage data
provided by Rocky Mountain Power. We next summarized the payment and arrearage data and the total
number of billing days for the pre- and post-periods for each account from one year prior to participation
through one year post-participation, based on the cost recovery date. We removed participant and comparison
group sites from our analysis if we did not receive at least 12 months of external payment and arrearage data
in the pre- or post-periods.
After applying the screening criterion, we were left with 121 participants and 48 comparison group clients out
of the original counts of 168 participants and 50 comparison group clients for the payment analysis. For the
arrearage analysis, we were left with 114 participants and 41 comparison group clients.
6.2 Results
Table 11 below presents the annual change in assistance payments annually and overall for the evaluation
period. Assistance payments decreased by an average of over 40% for Program participants while it increased
by over 60% for the comparison group. A net reduction in external payments of $112 is the net benefit of the
Program.
15 We intended to use the variable “measure effective date” but the program tracking data for participants in 2013 did not include
this variable. To remain consistent in our treatment of participants we relied on the “cost recovery date”, which was available for all
participants. The difference between the two date fields was, on average, less than one month, so we felt it would be close enough to
the date that measures were installed in participants’ homes. Cost recovery date is used as a proxy for measure installation date
throughout the payment analysis.
Payment and Arrearage Analyses for Non-Energy Benefits
opiniondynamics.com Page 29
Table 11. Payment Assistance Amounts Summary for Participants and Comparison Group
In addition to a reduction in external assistance payments, we examined the change in arrearages. An
arrearage is the unpaid ending monthly balance on a customer’s bill. To estimate this non-energy benefit, we
calculated the change in arrearage payments for Program participants and compared this to the change in
arrearage payments for the comparison group. Table 12 presents the findings from this analysis. The average
monthly arrearage for the participant group decreased by $5 while it increased by $12 for the comparison
group. The net difference is $17, however similar to the payment analysis above, the net difference does not
represent the non-energy benefit because the participant group nor the utility benefit from the increased
arrearages paid to the comparison group. The net Program benefit is the $5 reduction in monthly arrearages
paid to the participants of the Program.
Table 12. Arrearage Summary for Participant and Comparison Groups
Net
Difference
Pre Post Change %
Change Pre Post Change %
Change Amount
2013 $ 229 $ 128 $ (101)-44% $ 1,464 $ 1,460 $ (4)0% $ 97 $ 101
2014 $ 278 $ 128 $ (149)-54% $ 1,433 $ 2,354 $ 921 64% $ 1,071 $ 149
2015 $ 275 $ 189 $ (86)-31% $ 2,245 $ 4,976 $ 2,731 122% $ 2,817 $ 86
Total $ 260 $ 148 $ (112)-43% $1,714 $2,930 $1,216 62% $ 1,328 $ 112
Year
Participant Group Comparison Group Net
Program
Benefit
Net
Difference
Pre Post Change %
Change Pre Post Change %
Change Amount
Monthly Arrearage $ 38 $ 33 $ (5)-14% $ 28 $ 40 $ 12 43% $ 17 $ 5
Participant Group Arrearage Comparison Group Arrearage Net
Program
Benefit
Cost-Effectiveness
opiniondynamics.com Page 30
7. Cost-Effectiveness
This section presents the cost-effectiveness findings for Navigant’s analysis of the Idaho Low Income
Weatherization Program for program years 2013-2015. Navigant completed cost-effectiveness tests of the
Program using various approaches: PacifiCorp Total Resource Cost (PTRC) test, Total Resource Cost (TRC) test,
Utility Cost (UTC) test, Ratepayer Impact Measure (RIM) test, and the Participant Cost Test (PCT). Each scenario
is analyzed using modeled assumptions provided by PacifiCorp.
All scenarios utilize the following assumptions:
Avoided Costs: Navigant performed a custom analysis of calculating avoided costs by using the
Residential Whole House decrement cost and the Residential Cooling load shape. The decrements
values were populated using the 2013 PacifiCorp Integrated Resource Plan (IRP) for program years
2013-2014 and the 2015 PacifiCorp IRP for program year 2015.
Modeling Inputs: Navigant utilized program level savings provided by Opinion Dynamics and
administration costs provided by Rocky Mountain Power in the file LIW Evaluation Cost-effectiveness
Inputs V2.xlsx.
Non-Energy Benefits (NEBs): Navigant incorporated select NEBs including payment assistance and
arrearages, which were provided by Opinion Dynamics. The direct cost of health and safety repairs is
also included as a NEB and is quantified as a cost-offset to the program. Health and safety repair costs
are provided by Rocky Mountain Power.
Benefit/Cost Tests: Multiple benefit/cost tests are reported including; PacifiCorp Total Resource Cost
Test (PTRC), Total Resource Cost Test (TRC), Utility Cost Test (UCT), Rate Impact Test (RIM), and
Participant Cost Test (PCT).
The cost-effectiveness inputs are as follows:
Table 13. Low Income Weatherization Program Inputs
Parameter 2013 2014 2015
Discount Rate 6.88%6.88%6.66%
Residential Line Loss 11.47%11.47%11.47%
Residential Energy Rate ($/kWh)$0.10620 $0.10490 $0.10480
Inflation Rate¹1.90%1.90%1.90%
¹ Future rates determined using a 1.9% annual escalator.
Cost-Effectiveness
opiniondynamics.com Page 31
Table 14. Low Income Weatherization Program Annual Program Costs
Table 15. Low Income Weatherization Program Annual Program Savings
Table 16. Low Income Weatherization Program Non-Energy Benefits
Table 17. Non-Energy Benefit Adjustments
Non-Energy Benefit Perspective Adjusted
Payment Assistance PTRC, TRC
Arrearage PTRC, TRC, UCT, RIM
Health and Safety PTRC, TRC
Program Year Utility
Admin
Admin
Program
Delivery
Eval,
Marketing,
Prog Devel.
Incentives
Total
Utility
Costs
Gross
Customer
Costs
2013 $20,847 $17,866 $361 $164,667 $203,741 $0
2014 $16,455 $13,260 $1,688 $150,694 $182,097 $0
2015 $20,502 $16,697 $3,099 $215,356 $255,653 $0
2013-2015 $57,803 $47,823 $5,148 $530,717 $641,491 $0
Program Year
Gross
kWh
Savings
Realization
Rate
Adjusted
Gross kWh
Savings
Net to
Gross
Ratio
Net kWh
Savings
Measure
Life
2013 101,771 86%87,690 100%87,690 25
2014 52,320 93%48,585 100%48,585 25
2015 68,016 92%62,805 100%62,805 25
2013-2015 222,107 90%199,080 100%199,080 25
Program Year Payment
Assistance Arrearage Health and
Safety
Total Non-
Energy
Benefits
2013 $98,295 $15,442 $28,760 $142,496
2014 $54,461 $8,555 $31,575 $94,591
2015 $70,400 $11,060 $23,297 $104,756
2013-2015 $223,155 $35,057 $83,632 $341,843
Cost-Effectiveness
opiniondynamics.com Page 32
The benefit/cost ratios for each of the cost-effectiveness tests are presented in Table 19.
Table 18. Benefit/Cost Ratios - Low Income Weatherization
Table 19 provides the cost-effectiveness results for the combination of program years 2013 through 2015.
Table 19. LIW Program Level Cost-Effectiveness Results – PY2013-2015
Table 20, Table 21, and Table 22 provide the cost-effectiveness results for each individual program year.
Table 20. LIW Program Level Cost-Effectiveness Results – PY2013
Program Year PTRC TRC UCT RIM PCT
2013 1.23 1.17 0.63 0.4 n/a
2014 1.24 1.18 0.64 0.4 n/a
2015 1.22 1.17 0.63 0.4 n/a
2013-2015 1.23 1.17 0.63 0.4 n/a
Cost-Effectiveness Test Levelized
$/kWh Costs Benefits Net Benefits Benefit/Cost Ratio
Total Resource Cost Test (PTRC) +
Conservation Adder $0.20 $1,702,926 $2,095,326 $392,401 1.23
Total Resource Cost Test (TRC)
No Adder $0.20 $1,702,926 $1,998,072 $295,147 1.17
Utility Cost Test (UCT)$0.20 $1,702,926 $1,077,711 ($625,214)0.63
Rate Impact Test (RIM)$2,681,579 $1,077,711 ($1,603,867)0.4
Participant Cost Test (PCT)$0 $2,570,805 $2,570,805 n/a
Lifecycle Revenue Impacts ($/kWh)$0.00 $0.0000185833
Discounted Participant Payback (years)n/a n/a
Cost-Effectiveness Test Levelized
$/kWh Costs Benefits Net Benefits Benefit/Cost Ratio
Total Resource Cost Test (PTRC) +
Conservation Adder $0.20 $569,791 $698,442 $128,651 1.23
Total Resource Cost Test (TRC)
No Adder $0.20 $569,791 $666,024 $96,233 1.17
Utility Cost Test (UCT)$0.20 $569,791 $359,237 ($210,554)0.63
Rate Impact Test (RIM)$896,009 $359,237 ($536,772)0.4
Participant Cost Test (PCT)$0 $856,935 $856,935 n/a
Lifecycle Revenue Impacts ($/kWh)$0.00 $0.0000062497
Discounted Participant Payback (years)n/a
Cost-Effectiveness
opiniondynamics.com Page 33
Table 21. LIW Program Level Cost-Effectiveness Results – PY2014
Table 22. LIW Program Level Cost-Effectiveness Results – PY2015
Cost-Effectiveness Test Levelized
$/kWh Costs Benefits Net Benefits Benefit/Cost Ratio
Total Resource Cost Test (PTRC) +
Conservation Adder $0.20 $562,120 $698,442 $136,323 1.24
Total Resource Cost Test (TRC)
No Adder $0.20 $562,120 $666,024 $103,905 1.18
Utility Cost Test (UCT)$0.20 $562,120 $359,237 ($202,882)0.64
Rate Impact Test (RIM)$888,337 $359,237 ($529,100)0.4
Participant Cost Test (PCT)$0 $856,935 $856,935 n/a
Lifecycle Revenue Impacts ($/kWh)$0.00 $0.0000061251
Discounted Participant Payback (years)n/a
Cost-Effectiveness Test Levelized
$/kWh Costs Benefits Net Benefits Benefit/Cost Ratio
Total Resource Cost Test (PTRC) +
Conservation Adder $0.20 $571,015 $698,442 $127,427 1.22
Total Resource Cost Test (TRC)
No Adder $0.20 $571,015 $666,024 $95,009 1.17
Utility Cost Test (UCT)$0.20 $571,015 $359,237 ($211,778)0.63
Rate Impact Test (RIM)$897,233 $359,237 ($537,996)0.4
Participant Cost Test (PCT)$0 $856,935 $856,935 n/a
Lifecycle Revenue Impacts ($/kWh)$0.00 $0.0000061927
Discounted Participant Payback (years)n/a
Conclusions and Recommendations
opiniondynamics.com Page 34
8. Conclusions and Recommendations
Rocky Mountain Power is adhering to best practices by delivering the program through community-based
agencies. SEICAA and EICAP have served as Program implementers on behalf of Rocky Mountain Power for
years. It is a common practice for utilities to work with community action agencies to bring their energy
efficiency programs to low income households since these organizations generally have well-established
relationships with them already. Additionally, these agencies are knowledgeable about using funding from
utilities in combination with government funding to expand the reach of programs. SEICAA and EICAP both
demonstrate their understanding of program processes, requirements and funding mechanisms. Leveraging
these type of agencies is a best practice in low income weatherization programs. Rocky Mountain Power
should continue to use the same Program implementers moving forward.
SEICAA and EICAP are consistent in their delivery and are adhering to federal guidelines and best practices to
ensure cost-effective delivery; both mentioned using EA5, a USDOE approved software package to conduct
audits. Both agencies also mentioned that as part of the audit, they input 12 months of energy usage data to
arrive at a more accurate estimate of energy savings when they model the installation of energy measures.
This had not always been a standard practice and came about during this most recent evaluation cycle.
Auditors recommend measures based on USDOE Weatherization Assistance Program guidelines for
installation which requires a Savings-to-Investment ratio of 1.0 or greater, when funds from the government
or Rocky Mountain Power are used.
Participants continue to be highly satisfied with the Program, the application process and agency staff. The
Program is giving energy conservation education that allows it to go beyond measure savings with behavior
savings as well. Most participants recall this education, find it extremely helpful and many took some of the
recommended actions. This education may be contributing to the strong net savings per participant.
Rocky Mountain Power has tried to increase awareness about its funding of the program, given that the utility
provides 85% of the costs of measures installed in participants’ homes. Most participants cannot recall who
funds the Program and those that do often associate it with the agencies instead of the utility. Only 10% of
surveyed clients identified Rocky Mountain Power as the funding source. In 2015, Rocky Mountain Power
started to send letters and magnets to participants to thank clients for participating and to increase awareness
of the utilities’ role in the program. These efforts may help increase association of the Program with Rocky
Mountain Power over time but the Program may also consider branding the agency staff who conduct the
audits and installation services by wearing shirts with the Rocky Mountain Power name and logo.
Long waiting lists to receive weatherization services continue from one agency’s perspective, although that
agency could not decipher the Rocky Mountain Power waiting list versus other utilities. It may not be a huge
issue for Rocky Mountain Power clients given that 62% of survey respondents said the Program served them
within 3 months of applying. SEICAA noted that it served all Rocky Mountain Power clients that qualified and
still had remaining funds. The demand for services may be higher than what Rocky Mountain Power can
provide, particularly for EICAP. However, since EICAP exhausted their Program funding and SEICCA did not use
all of its funding, Rocky Mountain Power may revisit the funding levels to each agency and consider giving
more to EICAP and less to SEICCA.
Based on the consumption analysis, the net energy savings (1,185 kWh per participant) and realization rate
(90%) for the program are very strong. The savings per participant is 55% of what was reported in the previous
evaluation period (2010 through 2012). We believe this lower estimate stems from a difference in the
measure mix installed in low income homes. No furnaces were replaced during the 2013-2015 program years,
but a total of 16 furnaces were replaced during the 2010-2012 Program years. The savings will likely be
persistent for many years as most participants (81%) are homeowners and the measures installed have long
Conclusions and Recommendations
opiniondynamics.com Page 35
effective useful lives, such as insulation. In addition, the Program is inducing non-energy benefits as well,
including reducing bills, reducing the need for external payments, and increasing the comfort, safety and
aesthetics of the home.
The Evaluation Team supports the Program’s decision to switch from CFLs to LEDs in 2016. With EISA
legislation, CFLs are slowly becoming the norm, half of the participants already had CFLs in their homes prior
to participation and this is likely impacting the savings as some of these bulbs may not have replaced
incandescents.
Though the Program has been well received, it has had declining participation since 2012. The decline in
participation could be due to several factors, including market penetration amongst the eligible population or
the depletion of ARRA funding. We recommend that Rocky Mountain Power take a historical look at
participation amongst its low income population that likely has electric heat to determine how much of the
market has been penetrated thus far. This exercise could also help to identify and target households that have
not participated yet.
The Program could reduce costs if agencies can verify that a client has electric heat before visiting the home.
Clients have difficulty with correctly identifying whether their home uses electric heating. Currently, the
agencies rely on clients to tell them if they have electric heat and then verify it by visiting the home. We
recommend that Rocky Mountain Power coordinate the transfer of electric usage data to the agencies to help
them determine if a client has electric heat before visiting the home. The average electric consumption for low
income households with electric heat could help agencies determine if a client is in the general ballpark before
visiting the home.
Finally, the Program is struggling with an issue commonly found in low income weatherization programs
throughout the country, i.e., overcoming the structural barriers to installing weatherization measures. These
structural barriers are an issue impeding participation and cost-effectiveness. This issue is a quandary to most
utilities who need to allocate funds directly to energy saving improvements, for cost-effectiveness standards,
instead of structural and safety improvements that do not directly lead to energy savings. While other funding
sources can help, it often is not enough. For most utilities, this remains an unsolvable dilemma. However, one
electric cooperative in Arkansas advocated for a new tariff in the state that allowed for an innovative financing
solution that directly solved this issue. The Pay-As-You-Save model, allows the utility to fund both structural
and energy improvements and provides immediate net savings for the client. The client does not incur a debt
obligation while the utility benefits from a low risk path to cost recovery through a charge on the bill that is less
than the estimated savings from the upgrades. We recommend that Rocky Mountain Power staff explore this
innovating financing tariff that allowed a utility to address both structural and energy improvements through
its low income weatherization program at no up-front cost to the client. More information on this innovate tariff
and how the program operates can be found in Appendix B.
Appendix A: Alternative Model Specifications
opiniondynamics.com Page 36
Appendix A: Alternative Model Specifications
We built alternative models to test the consistency of the savings estimate from the basic model. We built our
first set of alternative models to look at the impact of using different algorithms for selecting the matched
comparison group. Comparison Group Alternative 1 took the one best match for each participant rather than
the top three matches. Comparison Group Alternative 2 continued to use the top three matches, but only
selected the match if the weather station area was the same for both the participant and the match. We show
very little variation in estimated savings using these alternative comparison groups, so we have confidence in
the results developed using the base model. We show very little variation in estimated savings using the
alternative models, as shown in Table 23.
Table 23. Model Results for Different Comparison Group Specifications
Model Post Variable
Coefficient
Annual KWH
Savings
Basic Model -3.24594 1,185
Control Group Alternative 1 -3.23580 1,181
Control Group Alternative 2 -3.26272 1,191
We built another set of alternative models to explore the impact of weather on the model results. While we did
weather-normalize the basic model by including HDD and CDD factors, our review of the data shows that there
was a significant difference in weather conditions between the pre- and the post- periods during the study
timeframe. We demonstrate this by calculating the percentage differences in the pre- and post-period average
annual heating and cooling degree days, as Table 24 shows.
Table 24. Difference in Weather Temperatures in Pre- and Post- Period
Model Pre- Period Post- Period Percent
Difference
Average Annual HDD 7,563 6,647 -12%
Average Annual CDD 420 387 -8%
It is possible that the warmer winters and cooler summers that occurred after installation of measures is
affecting the impact estimates beyond what the basic weather-normalization model can account for.
Another indication that this may be an issue is the fact that the coefficient on the MonthIndex variable in the
basic model is negative, indicating a small, continuous decrease in usage across all clients during the study
time frame. We find this unusual because the coefficient on the time series variable is often positive in other
consumption analyses, reflecting the fact that there is a small increase in usage over time across all clients
as they add electric end uses into their lifestyle. We believe it is possible that the MonthIndex variable is
picking up some of the decrease in usage that is actually a result of the milder weather that occurred in the
later years of the study.
We tested four alternative models to see if they could do a better job of identifying Program-induced savings
during this time of increasing mildness in the weather. We present the results of each of these alternative
models in Table 25 below.
We added separate variables related specifically to HDD and CDD conditions during the post period in the
Weather Alternative 1 model. By doing so, we theoretically created weather-normalized savings estimates
based on the weather that occurred during the post period. Our results show that participants increased their
summer usage (presumably air-conditioning) rather than decreased it in the post period. This increased
Appendix A: Alternative Model Specifications
opiniondynamics.com Page 37
summer usage which offset the savings seen in the winter and in year-round base usage, creating an overall
annual savings estimate similar to the basic model results. However, this increase in summer usage does not
make much sense. It is possible that participants started using air-conditioning more after their homes were
weatherized because it became more affordable, but this is unlikely given the fact that the summers were
comparably much milder in the post period.
Table 25. Comparison of Model Results for Different Weather Specifications
Model
AIC
Month
Variable
Coefficient
Base
Annual kWh
Savings
(Based on
Post
Coefficient)
HDD
Annual kWh
Savings
(Based on
Post*HDD
Coefficient)
CDD
Annual
kWh
Savings
(Based on
Post*CDD
Coefficient)
Annual
KWH
Savings
Basic Model 46,405 -0.04512 1,185
Weather Alternative 1 46,368 -0.04899 834 564 (238) 1,159
Weather Alternative 2 46,375 1,355 537 (241) 1,631
Weather Alternative 3 46,374 -0.04952 277 911 0 1,188
Weather Alternative 4 46,603 -0.06474 482 772 0 1,254
In the Weather Alternative 2 model, we keep the new Post*HDD and Post*CDD variables, but remove the
Month variable to test if Month is actually reflecting program-induced savings. Our results show that the
decreasing usage the Month variable picked up in the Weather Alternative 1 model gets shifted to the post
variables, creating an Annual kWh savings estimate of 1,631 kWh per client per year, which is much higher
than the 1,185 kWh of savings from the Basic Model. However, this new model specification changes very
little in the estimate of savings that come from space heat or cooling. It all goes to base usage. Since it is hard
to justify why this program would impact base usage instead of space heat or cooling usage, and the Month
coefficient is statistically significant for the combination of all pre- and post- and participant and comparison
group observations, we hypothesize that the Month variable is truly picking up a non-Program-related trend
and therefore should be retained in the model.
In the Weather Alternative 3 model, we put back the MonthIndex but drop Post*CDD since it has a coefficient
with the wrong sign. We see savings estimates very similar to the basic model, indicating that the influence of
the CDD coefficient is not really a significant factor in the overall estimate of program savings.
Weather Alternative 4 goes one step further and keeps MonthIndex but drops both CDD and Post*CDD to
check if the CDD effect has any influence on the weather-normalization of the model at all because this area
has such low air-conditioning need. We see a small increase in annual Program savings because there is no
accounting for the fact that milder weather in the post period created some reduction in usage outside of the
program.
While the negative coefficient on the MonthIndex variable is still a bit perplexing, none of the alternatives did
a better job of accounting for milder weather in the post period. We therefore recommend that the basic model
be kept as the best estimate of program-induced savings for the Program. This is true even though the AIC is
slightly lower for some of the alternative weather models. We feel the greater transparency and ease of use
related to the simplest Basic Model is more useful than the complications that occur from alternative models.
We also see very little difference in results from selecting the simplest model.
Appendix B: Alternative Financing Documentation
opiniondynamics.com Page 38
Appendix B: Alternative Financing Documentation
Arkansas Pay as You Save Tariff
Appendix B: Alternative Financing Documentation
opiniondynamics.com Page 39
Appendix B: Alternative Financing Documentation
opiniondynamics.com Page 40
Appendix B: Alternative Financing Documentation
opiniondynamics.com Page 41
Ouachita Electric HELP PAYS Program
Appendix B: Alternative Financing Documentation
opiniondynamics.com Page 42
Appendix B: Alternative Financing Documentation
opiniondynamics.com Page 43
Appendix B: Alternative Financing Documentation
opiniondynamics.com Page 44
Appendix B: Alternative Financing Documentation
opiniondynamics.com Page 45
Appendix B: Alternative Financing Documentation
opiniondynamics.com Page 46
Appendix B: Alternative Financing Documentation
opiniondynamics.com Page 47
Appendix B: Alternative Financing Documentation
opiniondynamics.com Page 48
Appendix B: Alternative Financing Documentation
opiniondynamics.com Page 49
Appendix B: Alternative Financing Documentation
opiniondynamics.com Page 50
Appendix B: Alternative Financing Documentation
opiniondynamics.com Page 51
Appendix B: Alternative Financing Documentation
opiniondynamics.com Page 52
Appendix B: Alternative Financing Documentation
opiniondynamics.com Page 53
For more information, please contact:
Aaiysha Khursheed, Ph.D.
Principal Consultant
858 401 7638 tel
858 270 5011 fax
akhursheed@opiniondynamics.com
7590 Fay Street, Suite 406
La Jolla, CA 92037