20140812Khawaja Exhibits 3.pdf

E
ldaho Electric
ME
Avista 2OL3
lmpact Evaluation Report
)une 17 ,2074
Avista Corporation
1411 E Mission Avenue
Spokane, WA 99252
The Cadmus Group, lnc.
An Employee Owned Company . www.(admu59roup..om
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 1, Page 1 of 130
This page left blank.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 1, Page 2ot 130
Prepared by:
Danielle C6t6-Schiff Kolp, MESM
Andrew Wood
Jeff Cropp
scott Reeves
Jim Stewart
Matei Perussi
Michael Visser
Andrew Reitz
Madison Busker
Zachary Horvath
M. Sami Khawaja, Ph.D.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 1, Page 3 of 130
This page left blank.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule l, Page 4 of l30
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 1, Page 5 of 130
5.3 Net Portfolio Savings
5.4 IRP Goals Achievement................... ..................................108
Appendix A: Residential Billing Analysis Model Specifications...................... ............... 109
Appendix B: Residential Behavior Program Data Cleaning Procedures............... .........112
Appendix C: Residential Behavior Program Regression Model Estimates. ...,......,........114
Appendix D: Low-lncome Weatherization Participant Survey
Appendix E: Low-lncome Weatherization - Billing Analysis Model Specification.................................... 123
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedulel, Page6of130
Definitions
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 1, Page 7 of 130
Reported Savings Electricitv savincs that are reoorted in Avista's trackins database-
Gross Evaluated
Savinr
Electricity savings that have been verified through evaluation activities such as
records review, verification survevs or site visits, and engineering analysis.
Reallzation Rate The ratio of gross evaluated savings over the reported savings.
Net Evaluated
Savings
The portion of savings directly attributable to the program; savings that would
have othenrvise not occurred without program influence. These also include
oarticiDant and nonparticioant sDillover.
Net-to-Gross Ratio Ratio of net evaluated savings to sross evaluated savinss.
SavinIs Goal lntesrated Resource Plannina or Avista Business Plan savinss soal
Achievement Rate Ratio of evaluated savings over the savings goal,
Portfolio Executive 5ummary
For several decades, Avista corporation has been administering demand-side management (DSM)
programs to reduce electricity and natural gas energy use for its portfolio of customers. Avista
contracted with cadmus to complete process and impact evaluations of the company's program year
(PY) 2013 electric DSM programs in ldaho; this report presents our impact findings.
Evoluotion Adivities
We conducted the evaluation using a variety of methods and activities shown in Table 1.
Residential
Space and Water
ConversionsL
] GeosraphiccFL
I Glveaway
i s;ii;io, p,.oeii;i
Savings Results
Overall, the ldaho portfolio achieved a 102.7% realization rate, and acquired 25,899,345 kWh in annual
gross savings (Table 2).
Table 1. PY 2012-PY 2013 Electric Programs'Evaluation Activities
Exhibit 3
Case Nos. AVU-E-14 AVU-G-l4
S. Khawaja, The Cadmus Group, lnc
Schedule 1, Page I of 130
Table 2. PY 2013 Reported and Gross Evaluated Savings
Residential
Nonresidential
Residential
Nonrisidential
io* ln.o."
Total
5,130,s07
77,602,253
&804,102
16,595,342
499,901
ls,ggg,rcs
15,595,342
4gg,gor
8,063,080
13,436,118
499,90L
21,999,099
5,933,L97 7L5.6%
943%
Low lncome 292,767 499,901 t70.8%
Residential Behavior' 2,L94,322 2,870,905 130.8%
Total 25,2Lg,Ug 25,E99,34'5 ,L02.7%
* Note that residential Behavior Program savings are inherently caiculited ai rrefand are therefore presented
here as net.
The overall net to gross ratio was estimated at 85% leading to 21,999,099 kWh of net savings (Table 3).
Goal Achievement
Table 4 and Table 5 show achieved savings toward the IRP and Avista Business Plan goals. Both goals
were exceeded. The IRP goal is set at the portlo lio-level. ln order to conduct sector-level analysis,
Cadmus adopted the Avista Business Plan goals by sector, and applied the corresponding proportions to
the IRP targets, The tables also show savinS achievements for the portfolio excluding the residential
Behavior program. The IRP goal is still met, but the more aggressive Business Plan goal falls short.
7,597,OO9 8,063,080
fg,aeO,ffe10,849,695
462,495
19,flr9,200 21,999,099
ld,iis,$qt
19,009,200
Table 3. 2013 ldaho Net Savings
Table 4. PY 2013 IRP Goals and Achieved Savings
S.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
Khawaja, The Cadmus Group, lnc
Schedule 1, Page 9 of 130
Table 5. PY 2013 Avista Business Plan Goals and Achieved Savings
Residential
Nonresidentiai
Low lncome
Total
Excluding Residential Behavior
8,547,340
iioqs,ez
513,589
21,109,2s1
2!,tog,25l
8,053,080
13,435,118
499,901
2t,999,O99
rg,r28,rg4
94.3%
111.5%
97.3%
104.2%
go.6%
Key Findings ond Conclusions
Portfolio Level
As shown in Figure 1, realization rates have remained fairly steady for the nonresidential sector and
increased over the last several years for residential and low income.
The national environment for DSM is becoming more challenging with the implementation of the Energy
lndependence and Security Act of 2007 (EISA), and more stringent codes and standards. Avista is
meeting these challenges with new measure and program ideas. On the residential side, light-emitting
diodes (LEDs) have been added to their upstream lighting program. For the nonresidential portfolio in
2014, Avista is starting a large fleet engine block heater program, targeting gas station canopy LED
lighting, and an exterior LED signage program.
Figure 1. Realization Rates of Portfolio Savings
t80%
L60%
t40%
L20%
700%
80%
50%
40%
20%
o%
Residential Nonresidential Low lncome
r2010-2011 .20L2 .",2013
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
Khawaja, The Cadmus Group, lnc
Schedule 1, Page 10 of 130
S.
ln future years, Avista may consider devoting additional resources to investigate new technologies and
program offerings. Some initial examples include the following:
. Home Performance with ENERGY STAR;
htto://www.enerqvstar.sovlindex.cfm?fuseaction=howes profiles.showsolash.
. Central air conditioners for residential application (as our general population research supports
a sizable load with customer stated intentions of potential increased saturations),
o A refresh of commercial direct install measures (either new, or repeat of measures installed 5-10
years ago),
. lnvestigate the upcoming Ienont Stor for leased commercial space,
. Commercial retrocommissioning or continuous commissioning (primarily for larger, complex
facilities such as hospitals and college campuses; for example,
htto://www.oge.com/en/mvbusiness/save/rebates/retrocommissioning/index.oage),
. Comprehensive compressed air system audits and upgrades to address both demand and
supply-side operation (based on Compressed Air Challenge best practices;
htto://www.comoressedairchallenee.orsl),
. Strategic energy management (similar to Energy Trust of Oregon's SEM program;
http://enerqwrust.orsllibrarv/Getoocument/1876).
Residential
For PY 2013, Avista's residential electric programs produced 8,053,080 kwh in net savings, yielding a
120% overall realization rate of reported savings, and 105% of equivalent residential IRP goals.
o Overall, residential electric customers responded well to the programs, often installing several
measures within the same year.
. Tracking databases proved adequate for evaluation purposes, providing sufficient contact
information and measure and savings information. During the database review, Cadmus
confirmed the information was reliable and accurate.
o All rebated measures had been installed and continued to operate.
. Homes participating in the Behavior Program saved on average 0.674 kwh (1.57%) per day. The
percentage savings were higher than expected (1.2%).
Nonresidential
For PY 2013, Avista's nonresidential electric programs produced L3,435,Lf8 kWh in net savings, yielding
a 94% overall realization rate of reported savings, and 124% of equivalent nonresidential IRP goals.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule '1, Page 11 of 130
Cadmus evaluated L42 of 6,476 measures installed through the programs, representing 16% of reported
savings. ln general, Cadmus determined that Avista implemented the programs well. Cadmus identified
the following key issues that led to adjusted enerSy savings:
Metering on post-installation power consumption for several industrial process measures
indicated that the evaluated energy savings varied from the reported value.
Some participants did not operate the incented equipment correctly or did not complete the
improvements expected for the measure.
Some participant post-installation heating or cooling loads did not achieve the level of projected
consumption, which reduced energy savings.
Simulation models sometimes did not accurately represent the actual as-built building or system
operation.
There were instances where thorough analysis of energy-savings calculations provided by
participants or third-party contractors was lacking.
Some projects had data entry errors in characterizing building or measure performance.
Low lncome
For PY 2013, Avista's low-income electric programs produced 499,901 kWh in net savings, yielding a
171% overall realization rate of reported savings and 108% of equivalent low income IRP goals.
Compared to PY 2010, Avista's PY 2013 low-income program demonstrated an increase in average
electric savings per participant, in addition to an increase in the overall program realization rate. Several
factors may have contributed to the increase in participant savings, including:
o An increased frequency of installing high-saving measures (e.g., shell), and
. changes in agency delivery protocols or energy-saving installations made with non-utility
funding.
One factor contributing to higher realization rates are lower average reported savings occurring in the
evaluation period compared to previous years.
Recommendations and Further Anolysis
Residential
Cadmus recommends the following changes to Avista's residential electric programs:
Consider updating per-unit assumptions of recycled equipment to reflect the findings in this
evaluation,
lf clothes washer rebates are ever reinstated, Avista should continue to track them all within the
electric program unless there is a large increase in penetration of gas dryers.
S.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
Khawaja, The Cadmus Group, lnc
Schedule 1, Page 12of 130
lncrease measure level detail capture on applications. Specific additional information should
include energy factors or model numbers for appliances, baseline information for insulation, and
home square footage, particularly for the ENERGY STAR Homes.
Consider tiered incentives by rating as higher SEER systems generally require EcM fan motors.
Consider completing a lighting logger study within its territory if Avista believes the results of
the forthcoming Residential Building Stock Assessment (RBSA) study do not accurately represent
usage in their territory.
Consider researching the percentage of Simple Steps, Smart Savings bulb purchase that are
installed in commercial settings. This will increase the average installed hours of use and
increase estimated program savings.
Perform a billing analysis on ENERGY STAR homes using a non-participant comparison group
once enough homes have participated underthe new requirements.
Consider researching the current variable speed motor market activity to determine if this
measure should continue as a stand-alone rebate or be packaged with other equipment
purchases.
. Continue to promote efficiency programs in the Behavior Program energy reports, as the reports
increased both the rate of efficiency program participation and savings.
. Avista should consider performing additional research about the peak-coincident demand
savings from the behavior program
Nonresidential
We have the following recommendations for improving program energy-savings impacts and evaluation
effectiveness:
. Create a quality control system to double-check all projects with savings over 300,000 kwh.
. Avista may want to consider tracking and reporting demand reduction to better understand
measure load profiles and peak demand reduction opportunities.
. Update prescriptive measure assumptions and sources on a regular basis.
. Streamline file structure to enable reviewers more easily identify the latest documentation.
. Continue to perform follow-up measure confirmation and/or site visits on a random sample of
projects (at least 10%).
r Consider flagging sites for additional scrutiny when the paid invoice does not include installation
labor as it may indicate that the work was not yet performed,
. Avista may consider adding a flag to their tracking database to automatically detect potential
outliers (e.g., savings per dollar (kWh/S or therm/S)).
a
a
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 1, Page 13 of 130
. ln the case of redundant equipment, Avista may want to consider incenting pump projects
through the Site-Specific Program to more accurately characterize the equipment operating
hours.
. Avista may want to set minimum standards for modeling design guidelines. The EnergyTrust of
Oregon provides an example on their website.
Low lncome
Cadmus recommends the following enhancements in order to improve program impact results:
o Consider including a control/comparison group in future billing analyses.
. Consider options for increasing the analysis sample size due to small program populations (such
as combining Washington and ldaho program participants).
. obtain a full list ofweatherization measures from agencies.
. Consider targeting high-use customers.
e Track and compile additional data from agency audits.
o Consider performing quantitative, non-energy benefit analyses.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 1, Page 14 of 130
1. Residential lmpact Evaluation
7,7. lntroduction
We designed our impact evaluation to verify reported program participation and energy savings. We
used data collected and reported in the tracking database, online application forms, phone surveys,
billing analyses, RTF savings review, and applicable updated deemed savings values.
7.2. Methodology
1.2.1. Sampling
Record Review Sampling
To determine the percentage of measures incented that qualified for the program, Cadmus designed
sample sizes to yield result at the 90% level of confidence and t10% precision level for each application
type, across both states and both fuel types. Cadmus randomly selected participant measures for a
record qualification review from the 2013 gas and electric program populations across both states
served. We sampled participants using a single measure record. However, if a customer applied for
multiple rebates on the same application form during the program year, we checked all measures
included in the application for qualification, whether the fuel was electric or gas.
Table 5 shows the number of record reviews we completed of unique accounts and unique measures.
ENERGY STAR Products
Home lmprovement
ENERGY STAR Homes
Survey Sompling
Cadmus conducted the participating customer surveys in February 2014. Table 7 provides a summary of
unique customers (identified using Avista account number) and surveys completed in each effort.
99
LO2
18
135
L42
is
Table 5. Measure-Level Record Reviews Completed
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 1, Page 1 5 of 1 30
Table 7, Residential Participant Details and Survey Sample-Combined Washington and ldaho
Itetu?al Gas and Electrlc Profams
Hlatins gnq coglinq !Igi"n"v
Water Heating
2,490 70
@316
3%l
r9x
Weatherization and Shell Measures 60 t9%
.rl r*l
37 24%
357 7%
313
Electrlc-only Programs
iicona n*rii.ratorina riieier n".v.tiru . 1,3t9 t
156
i 5,?76
Space and water Conversions
Total
Cadmus designed participant survey completion targets to yield results with 90% confidence and t10%
precision levels at the measure-category and state level. Cadmus deemed this necessary as data
collected through these surveys-specifically installation rates-were used to inform an impact
assessment of Avista's residential programs. The participant survey sampling plan also drew upon
multiple factors, including feasibility of reaching customers, program participant populations, and
research topics of interest.
Cadmus did not conduct participant surveys with Simple Steps, Smart Savings customers, as that
program has an upstream focus and therefore does not track participant contact information. Similarly,
for ENERGY STAR Homes, Cadmus did not survey residential customers purchasing rebated homes as the
rebates were paid to the builders. Cadmus also did not survey Residential Behavior program
participants.
Within each program, Cadmus randomly selected program participant contacts included in survey
sample frames. A review of collected data shows geographic distribution of survey respondents
clustered around urban centers, specifically the cities of Spokane, Coeur d'Alene, Pullman, Moscow, and
Lewiston. This alitns with population distributions in Avista's service territory. Figure 2 provides the
distribution of participating customer survey respondents.
70
S.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
Khawaja, The Cadmus Group, lnc
Schedule 1, Page 16 of 130
E@
Figure 2. Geographic Distribution of PY 2012 - PY 2013 Participating Customer Survey Respondents
(t.,r,.l
,io
a
. l.;,a:
,.
o."
1.2.2. Data Collection and Analysis
Record Review
Cadmus reviewed all records for the selected sample of accounts, checking them for completeness and
program compliance using the data they contained. Measures qualified if all data found in the
application complied with the program specifications. As Cadmus randomly sampled customers by
application type (and several measures can be found on different application forms), we tracked
qualification rates by the type of application. All 2013 sampled applications qualified for program
Incentives.
Surveys
Cadmus contracted with market-research firm Discovery Research Group (DRG) to conduct surveys with
the selected participants. To minimize response bias, DRG called customers during various hours ofthe
day and evening, as well as on weekends, and made multiple attempts to contact selected participants.
Cadmus monitored survey phone calls to ensure accuracy, professionalism, and objectivity. We analyzed
the survey data at the program level, rather than at the measure level. Survey results at the portfolio
level are weighted by program participation to ensure proper representation.
o
11
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
Khawaja, The Cadmus Group, lnc
Schedule 1, Page 17 of 130
S.
Databose Anolysis
Cadmus reviewed the participant database provided by Avista to check for inconsistencies in reported
savings and measure duplications. This review is necessary as Avista uses the database to track both
achieved savings and rebates paid. Our review revealed multiple cases for the tracked savings did not
follow the 2012 Avista TRM. These differences are described later in the report.
Unit Energy Sovings
Cadmus reviewed every high impact prescriptive measure except the weatherization and shell measures
for which we determined savings from a billing analysis. During each program year, Avista updates unit
energy savings (UES) to reflect the gross energy savings achieved by a measure's installation. Details on
each measure are included in the program sections below,
Billing Analysis
Cadmus conducted a statistical billing analysis of monthly meter data to determine the adjusted gross
savings and realization rates for the following electric measures: weatherization, conversions to air
source heat pump, and conversions to natural gas, We used a pre- and post-installation combined
Conditional Savings Analysis (CSA) and Princeton Score Keeping Method (PRISM) approach.
Verilicotion Rotes
Cadmus determined verification rates for each program. Where applicable, we administered verification
site visits and surveys, which included:
o Checking correct measures were tracked in the database;
. Correct quantities were accounted for; and
. Units remained in place and were operable.
1.2.3. Measure Qualification Rates
Cadmus considered a measure qualified if it met the requirements in its category such as being ENERGY
STAR-certified or meeting the minimum efficiency standards for the program. We ensured all
qualifications were met and, when necessary, conducted online database searches of the model
numbers and noted qualif,ing characteristics. All measures reviewed qualified for program incentives.
The total qualification rate for all 2013 residential electric programs was therefore 10096.
1.i. Program Results and Findings
1.3.1. Overview
Cadmus analyzed data records, maintained by either Avista or an implementation contractor, to
determine appropriate unit energy savings (UES) and measure counts for each supported measure
within each program. The end result is the total adjusted gross savings for each measure and program,
as well as the overall realized savings for each program.
72
Exhibit 3
Case Nos. AVU-E-l4 AVU-G-l4
S. Khawaja, The Cadmus Group, lnc
Schedule 1, Page 18 of 130
We followed the same steps for calculatinB adjusted gross measure savings for all programs except
Simple Steps, Smart Savings, Second Refrigerator and Freezer Reclcling, and Residential Weatherization:
Review program database to determine if the adjusted measure counts correctly represent the
number of installations.
Conduct a phone survey or site visit to verify that the installation is within Avista's service
territory.
. Calculate verification and qualification rates.
. Calculate deemed measure savings for products rebated during the program period.
. Apply verification and qualiflcation rates and deemed savings to the measure counts to
determine the adjusted gross savings for each measure.
Details on the calculation methods used for Simple Steps, Smart Savings*, Second Refrigerator and
Freezer Rerycling, and Residential Weatherization are included in their specific sections below.
1.3.2. Simple Steps, Smart Saving
Program Description
Avista's Simple Steps, Smart Savings is an upstream incentive program that is an effective alternative to
traditional mail-in incentives because of its ease of participation, widespread accessibility, and low
administrative costs. This type of program allows utilities' incentives to pass directly from manufacturers
to retailers, which then reduce bulb prices to their customers. The program motivates retailer
participation by reducing bulb prices without a loss in profits. For the customer, participation may be so
seamless they are unaware they have purchased an incentivized bulb or participated in a utility
program.
Upstream programs, however, pose particular evaluation challenges because calculating metrics, such
as in-service rates (lSR) and attributions, traditionally relies on surveying purchasers of incentivized
products. As part of our determination of program savin8s, we referred to the Northwest Regional
Technical Forum (RTF) UES assumptions, Avista's program records, and metering data collected by
Cadmus for similar measure installations.
This program incents various CFL5 and LEDS from standard twist to specialty bulbs that include 3-way,
reflector, dimmable, globe, and other specialty bulbs. There are unique assumptions for standard twist
bulbs and specialty bulbs; therefore, each was analyzed separately. Based on program funding,30% of
all bulb sales are assumed to be associated with residential sockets in ldaho.
13
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
Khawaja, The Cadmus Group, lnc
Schedule 1, Page 19 of 130
S.
Anolysis
This program has six different parameters to inform the calculation of gross savings for the lighting
component: CFL wattage, delta watt multiplier (DWM), hours-of-use (HOU), days-per-year, waste heat
factor (WHF), and lSR. The following algorithm shows the annual energy lighting savings:
Measure Watts = Wattage of the purchased CFL or LED
DWM = The difference in wattage between the baseline bulb and the
measure bulb divided by the wattage of the measure bulb
= Daily lighting operating hours
= Days peryear,365.25
= An adjustment representing the interactive effects of lighting
measures on heating and cooling equipment operation
= ln-service rate, or percentage of units installed
Specialty
LED Bulb
35,552I s,aae
39,583
9,446
9
179l!5
LED Fixture 9
Total t74,068
Totols moy dilfer lrcm the sum of volues due to rcunding.
14
@ @ @,@ @ @ @
Where:
HOU
DAYS
WHF
rsR
The annual savings algorithm is derived from industry-standard engineering practices, consistent with
the methodology used by the RTF for calculating energy use and savings for residential lighting. Each
methodology component is discussed in detail below.
CFL Wattage
Table 8 shows the reported and evaluated bulb and fixture sales forthis program. Evaluated sales were
determined from vendor provided data documenting sales allocated to Avista's territory. This
discrepancy is likely due to monthly adjustments made in the database, which in turn may have led to
either an over- or under-counting of the total sales volume.
Table 8. Total Reported and Evaluated CFLs Sold by Year
S.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
Khawaja, The Cadmus Group, lnc
Schedule 1, Page 20 of 130
Avista sales data included CFL wattage, units sold, and bulb type. Savings for each bulb type ls analyzed
separately. For 3-way bulbs, the middle wattage was used for the analysis. The average weighted CFL
wattage sold in PY 2013, for standard twist, specialty, LED bulb, and LED fixture, was 15.15 watts, 14.23
watts, 10.19 watts, and 13.94 watts, respectively.
Delta watt Multiplier
Cadmus followed the lumens equivalence method as laid out in the Uniform Methods Project (UMP) to
evaluate the baseline wattage and the DWM for each wattage and type of bulb sold. The evaluation
team matched the reported SKU numbers against the ENERGY STAR lighting databaser to determine the
lumens associated with each bulb. once the lumens value was determined, the baseline wattage was
evaluated in accordance with the guidelines outlined in the Energy lndependence and Security Act (EISA)
of 2@7.
ln PY 2013, Cadmus was able to match 83.1% of the roughly 500,000 bulbs incented through the
program, For the remaining 15.9% of bulbs, we determined the lumens value with an interpolation
equation that is based on the relationship between CFL wattage and lumen output from the ENERGY
STAR lighting database:
CFL Lumens in PY 20\3 = 70.952 x CFLWattage - 86.7I
Figure 3 and Figure 4 show a comparison of the lumens determined by lookup to the lumens determined
by regression model, along with the PY 2013 sales data for the given wattage. The figures shows that the
regression equation used in PY 2013 is a good estimate of the lumens output for a given measure
wattage, especially considering the low percentage oftotal program sales. Cadmus accepted the lumen
output estimated by the regression for both types of bulbs due to the low percentage of sales volume
used in the regression analysis.
75
S.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
Khawaja, The Cadmus Group, lnc
Schedule '1, Page 2'l of 130
30%g
24% 3
Eo18% boAt2% Z
c6%EoA
@6
5,0O0
4,m0
E 3,ooooE3 2,000
1,000
0
I
,///v,--<- r.. - I .
8 9 10 13 14 1s 18 19 20 21 22 23 26 32 40 42 55 68
CFL Wattate
r % of Sales, tumens by Lookup 'r, . .; % of Sales, Lumens by Regression
-Lumens
by Lookup
-Lumens
by Regression
Figure 4. Results of PY 2013 Lumens Determination, Specialty CFLs
L5%
g
LZ% 3
E69%tsIE6%t
C?%goA
o%
2,500
2,000
E 1,s00o
E
.= 1,ooo
s00
0
AI /-./ LA
_/-\,/ \
E<
--'r t
7 9 Lt t2 L3 L4 1s 16 18 t9 20 2L 22 23 24 25 26
CFL Wattage
r % of Sales, Lumens by Lookup -,lrrj % of Sales, Lumens by Regression
-Lumens
by Lookup
-Lumens
by Regression
Cadmus then determined the baseline wattage for each bulb based on the lumen output and whether
the bulb includes a reflector (which is not impacted by ElsA).'zTable 9 and Table 10 show the schedules
Cadmus used to determine the baseline wattage, for reflector and non-reflector bulbs, respectively. We
then calculated the DWM for each bulb using the baseline wattage and purchased CFL wattage.
2 Federal exemptions for some reflector-style bulbs were set to expire in late 2012. ln order to maintain
consistency between this evaluation and the PY 2012 evaluation, Cadmus assumed that the exemptions
expired on January 1, 2014. These exemptions would have caused a 0.69% decrease in overall PY 2013 savings.
76
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 1, Page 22ol 130
0-309
310 -74g
750 - 1,049
L,490 - 2,600
z,oor - s,eoo
e,gor - c,g1s
E@
Table 9. 2013 Baseline Wattage Based on Measure Lumens, Non-Reflector Bulbs
0-419
+zo - soo
551 - 837
ase - r,io3
23.92
24.26
o.oo
t,zc4-\ee:
Lasz-z,Eias
i,tro -zots
25
40
50
53
72
150
2oo
0.00
9.55
13.43
18.85
23.27
4L.77
62.34
11.00
L3.24
0
75,356
283,365
47,596
96,976
954
593
509
1,050
0.0%
12.6%
47.6%
8.0%
ts,lx
0.2%
0.1%
0.1%
0.2%
13.0%
o.7%
30
45
77,335
i,ria
65
75
1.2%
o.2%
o.ox
90
L20rzi
14.82
16.65
6,943
1,013-
Hours-of-Use
Cadmus estimated standard twist CFL HOU for residential installations using Avista's survey of room
types and a multistate modeling approach, built on light logger data collected from five states: Missouri,
Michigan, Ohio, Maine, and Maryland.3 A regression statistical model calculated the average HOU, using
combined multistate, multiyear data. Cadmus used the multistate model's estimate of HOU by room
type, weighted based on Avista's survey results to determine an overall average HOU of 2.38.
Though the Simple Steps, Smart Savings* program could introduce bulbs into residential and
commercial applications, an alFresidential application presented the more conservative assumption. As
compelling evidence did not exist to assume a proportion of commercial sales, Cadmus exclusively used
residential assumptions in this analysis.
Waste Heat Factor
The WHF is used to account for the change in annual HVAC energy, either lost or gained, due to a
reduction in facility lighting energy. Cadmus based the WHF on SEEM building models, developed by the
3 The Cadmus G rcup, lnc. 2O7O Evdluotion, Meosurement, ond Verification Report. Preparedfor Dayton Powel
and Light. March 15,2011.
77
Table 10, 2013 Baseline Wattage based on Measure Lumens, Reflector Bulbs
S.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
Khawaja, The Cadmus Group, lnc
Schedule 1 , Page 23 of 1 30
Regional Technical Forum (RTF). These SEEM building models estimate the change in HVAC equipment
energy use resulting from a change in lighting technology (e.g., from incandescent lamps to CFLs). ln
general, the models account for the interaction using load shape profiles of the HVAC and lighting
equipment, based on dwelling occupancy.
The RTF uses an inherently conservative method, as it assumes a closed shell (i.e., all interior lamps),
including ceiling recessed cans contained in a closed system. Thus, heat produced by the bulbs enters
the building. ln reality, waste heat could transfer out of the conditioned space.
Cadmus based the calculation on Avista's share of electric heating equipment a along with its associated
efficiencies and surveys of interior and exterior distributions, producing a WHF of 89.8%.s
ln-Service Rate
Cadmus used the same CFL ISR accepted end approved by the RTF of 74.48%.5 This a storage rate of 24%
and a removal rate of 2%. The Council's method to determining ISR is inherently conservative, because it
assumes that the remaining 24% of bulbs in storage never provide energy savings. However, research
has revealed that almost all program bulbs are installed within three years of purchase. Cadmus used
the same LED ISR accepted and approved by the RTF of 1OO%.7
Results and Findings
Overall Program Savings
Avista's total reported and evaluated savings for PY 2013 are shown in Table 11,
Avista equipment-type saturations derived from a 2011 participant survey for the Geographic CFL Giveaway
Program.
The default RTF WHF is 86.4%.
See: httD://rtf .nwcou ncjl.orelmeasures/measure.aso?id=142
18
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 1, Page 24 of 130
Twist
sftcialty
tED Bulb
LED Fixture
Total
aealiiitlon nate
128,960 . 3,095,050
35,552 588,258
9,446 196,366
a.7 209
17406E 3,879,883
24.O
15.5
20.8
24.O
22.t
26.5
2s.8
29.0
27i
26.5
t29,70't 3,437,438
39,583 L,O22,349
9,446 273,s72
8.7 240
t?a,743 AJ!,S,6OO
103%t799,
Totols may differfrom the sum oJvolues due to rounding.
Showerheads
Though primarily a li8hting program, Simple Steps, Smart Savings also incentivized low-flow, energy-
saving shower heads in PY 2013. The evaluation assumes that 52.1% ofthe units purchased were
installed in homes with an electric water heater and 47.9% of the units were installed in homes with a
gas water heater. This assumption is based on the responses of almost 400 of Avista's residential
customers in ldaho to Cadmus'general population survey. The program sold showerheads with flow
rates ranging from 1.5 gallons per minute (gpm) to 2.0 gpm. The unit energy savings for each flow rate
sold are based on the net savings values currently approved by the RTFs for showerheads purchased
through a "Retail" program and installed in "Any Showef in the home. Evaluated savings follow the RTF
methodology and include the electricity savings due to reduced water and sewer requirements for all
units purchased through the program. The assumptions used and unit energy savings (UES) calculated
for this evaluation are shown in Table 12.
8 htto://rtf.nwcouncil.orslmeasures/measure.asp?id=126
722%
79
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 1, Page 25ol 130
Table 12. Showerhead Assumptions
Unlts Sold
2013 showerheads sold
Su' ey Results, Fuel Dinribution
PercentcasDHw 47.9%
Percent Electric DHW 52.1%
2t2
--
2ol3ElectricwaterHeatersavings(kwt) i 139.2
ZOfS Gi Water tleater Savings (therms) S.Z
water & sewer savin$ - All units sold I uts
zbii water a serer srvings Uwfrl I 6..i
The total savings for these units are shown in Table 13. The Electric Savings per Unit Purchased shown in
the table apply to all units purchased through the program as it accounts for the saturation or electric
and gas equipment as well as the water and sewer savings.
Table 13. Simple Steps, Smart Savings, 2013 Showerhead Savings
Units Purchased
Program Savings (kWh)_?12._ _ _ _1ry}
!!,70q l_ - EryL2,344
Electric Savin8s Per Unit Purchased (kwh)58.2 78.8 135%
1.3.3. Second Refrigerator and Freezer Recycling
Summory of Program Porticipotion
Cadmus reviewed the participant database, maintained by JACO, the program implementer, to test the
reliability of program data. As shown in Table 14, the program rerycled 348 units during PY 2013, a slight
increase relative to PY 2012. Some participants recycled more than one appliance through the program.
20
S.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
Khawaja, The Cadmus Group, lnc
Schedule 1, Page 26 of 130
Iable 14. ldaho Program Participation by Measure
20to
201L
20L2
2013
Total
Recycled Refrigerator
Recycled Freezer
Total
n-y.l"d n"friger.tor
Recycled Freezer
Total
Recycled Refrigerator
n".y-l"d rreeret
Total
hecycled Refrigerator
Recycled Freezer
Total
R;cycled Refrigerator
Recycled Freezer
Total
317
75
,9t
4L2
L27
533
257
70
127
275
73
gaa
L,261
339
1,600
As shown in Figure 5, side-by-side refrigerators made up a larger percentage of program participation in
PY 2013 than in previous years. lncreasing quantities of side-by-side refrigerators is typical of maturing
appliance recycling programs. The proportion of bottom freezer units was unchanged from previous
years. The proportion of single door units decreased slightly since PY 2012.
Figure 5. Refrigerator Configurations by Program Year
1m%
90,6
a@5
70,6
60/n
s@6
4M
3Vo
2ffi
10,6
o%
r Eottom Freezet
Side{y-Side
r Single Door
! Top Freezer
PY 2010 PY 2011 N 2012 PY 2013(wA&lD) (wA&lD) (lDonly) (lDonly)
As shown in Figure 6, chest freezers made up a larger proportion oftotal participation in PY 2013 than in
PY 2072.
21
S.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
Khawaja, The Cadmus Group, lnc
Schedule 1, Page 27 ot 130
Figure 5, Freezer Configurations by Program Year
1@96
90,6
AM
7Vo
ffi6
50,5
40,5
3M
z(fr
106
w6
r Chest
r uprEht
PY 2010 PY 2011 PY 2012 PY 201:}
(wA & lD) (WA & lD) (lD only) (lD only)
ln 2013, recycled refrigerators averaged 28 years old, with 17.9 cubic feet of internal capacity. Recycled
freezers averaged 35 years old, with 18.6 cubic feet of internal capacity,
Determining Averoge Annuol Gross Sovings
Cadmus developed a multivariate regression model to estimate the gross savings of retired refrigerators
and freezers. We estimated the model coefficients using an aggregated in situ metering dataset
composed of over 500 appliances (which we metered as part of five California, Wisconsin, and Michigan
evaluations conducted between 2()()9 and 2012). These evaluations reflected a wide distribution of
appliance ages, sizes, configurations, usage scenarios (primary or secondary), and climate conditions.
UMP and RTF Protocols
Recent guidelines developed by the U.S. Department of Energy (DOE) informed Cadmus' impact
evaluation methodology for PY 2012 and PY 2013. ln 2011, DOE launched the UMP, intending to
"strengthen the credibility of energy savings determinations by improving EM&V, increasing the
consistency and transparency of how energy savings are determined."e
The UMP identifies seven common residential and commercial DSM measures, reporting results from an
enlisted set of subject matter experts who drafted evaluation protocols for each measure category,
Refrigerator recycling was one of the seven identified measures. The DOE recruited Cadmus to manage
the UMP process and to serve as the lead author for the refrigerator recycling protocol.
e U.S. Department of Energy. About the Unihrm Methods Pro.rect. Accesed April 24, 2014. Available online:
htto://energv.sovleere/about-us/uniform-methods-oroiect{etermininr-enerqv-efficiencv-orolram-
savinss/about-uniform-methods,
22
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 1, Page 28 of 130
Through a collaborative process that included reviews by a technical advisory group and a steering
committee, as well as a public review and response period, the UMP resulted in a set of protocols
capturing the collective consensus ofthe evaluation community. Each protocol establishes broadly
accepted best practices for evaluating key measures in that category, including methods for identifying
and explaining key parameters, data sources, and gross- and net-related algorithms.
This evaluation of the Avista's PY 2013 ARP in ldaho followed the complete UMP methodology outlined
in the refrigerator recycling protocol. The DOE websitelo provides more information about the UMP
Refrigerator Regression Model.
Ref rigerator Regression Model
Table 15 shows the variables we used to estimate refrigerators'annual energy consumption, along with
the estimated parameters.
Table 15. Refrigerator UEC Regression Model Estimates
(Dependent Variable = Average Daily kWh, R'z= 0.30)
lntercept
age (veirsl
Dummy: Manufactured Pre-1990
Size (cubic feei)
Dummy: Single Door
Dummy: Side-by-Side
Dummy: Primary
lnteraction: Unconditioned Space x HDDs
lnteraction: Unconditioned Space i CDDi
0.80s
0.021
1.036
0.059
4.75r
1.720
o.sao
-o.o4o
o.026
0.166
0.152
<.oooi
0.044
<.0001
..OOOf
o.ooe
o.oor
o.is8
The results of our analysis indicated the following:
o Older refrigerators experienced higher consumption due to year-on-year degradation.
. Refrigerators manufactured before the 1990 National Appliance Energy Conservation Act
(NAECA) standard consumed more energy.
. Larger refrigerators consumed more energy.
Single-door units consumed less energy, as these units typically did not have full freezers.
Side-by-side refrigeretors experienced higher consumption due to greater exposure to outside
air when opened and due to the through-door features common in these units.
Primary appliances experienced higher consumption due to increased usage.
U.5. Department of Energy. "Uniform Methods Project for Determining Energy Efficiency Program Savings."
Accessed April 24,2014. http://enersv.govleere/about-us/initiatives-and-oroiects/uniform-methods-proiect-
determining-energv-eff iciencv-Drogram-savinEs.
a
23
S.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
Khawaja, The Cadmus Group, lnc
Schedule '1, Page 29 of 130
a
a
At higher temperatures, refrigerators in unconditioned spaces consumed more energy.
At colder temperatures, refrigerators in unconditioned spaces consumed less energy.
Freezer Regr6sion Model
Table 15 shows the freezer model details.
lntercept
Table 15. Freezer UEC Regression Model Estimates
(Dependent Variable = Average Daily kWh, R-square = 0.38)
Age (years)0.045 0.001
Dummy: Manufactured Pre-1990
size (cubic feet)
Dummy: Primary
lnteraction: Unconditioned Space x XOOs
qur11'y: lhest Flejq 0,298 ;-0.031
0.082
0.002
0.2g2
<.0001
o.o-t
lnteraction: Unconditioned Space x CDDS o.237
The results of our analysis indicated the following:
o Older freezers experienced higher consumption due to year-on-year degradation.
. Freezers manufactured beforethe 1990 NAECAstandard consumed more energy.
. Larger freezers consumed more energy.
. Chest freezers experienced higher consumption,
. At higher temperaturet freezers in unconditioned spaces consumed more energy.
. At colder temperatures, freezers in unconditioned spaces consumed less energy.
Extrapolatlon
After estimating the final regression models, Cadmus analyzed the corresponding characteristics (the
independent variables) for participating appliances (as captured in the JACO database). Table 17
summarizes program averages for each independent variable.
24
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 1, Page 30 of 130
As an example, using values from Table 16 and Table 17, Cadmus calculated the estimated annual UEC
for PY 2013 freezers as:
2013 Freezer UEC =
355.25 days * (-0.955 f 0.045 + l34.SB years oldl + 0.543 + l79o/ounits manuf actured pre -
19901 + O.120 * 1L8.56 f t.3I + 0.298 * f36oh units that are chest freezersl * 0.082 *
10.43 (tnconditioned CDDsI - 0.031 * 19.47 llncond.itioned HDDsI) = t,t39 kWhlyeartl
Refri8erators
Age (years)
Dummy: Manufactured Pre-1990
Size (cubic feet)
Dummy: sintle Door
Dummy: side-by-5ide
Dummy: Primary
lnteraction: Unconditioned Space x Heating Degree Days
lnteraction: Unconditioned Space x Cooling Degree Days
Age (Years)
Dummy: Manufactured Pre-1990
Size (cubic feet)Freezers Dummy: Chest Freezer
lnteraction: Unconditioned Space x Heating Degree Days
lnteradion: Unconditioned Space x Cooling Oegree Days
Figure 7 compares distributions of estimated UEC values for refrigerators and freezers.
The UEC shown is hiSher than that calculated from the coefficients and means shown in the UEC equation,
which are rounded. Cadmus used unrounded coefficients and means for calculating the evaluated UEC.
25
28.33
o.72
17.85
o.o1
o.2L
o.40
6.66
o.s1
33.8t
o.7g
18.56
0.36
9.47
o.4t
Table 17. PY 2013 Participant Mean Explanatory Variables
S.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
Khawaja, The Cadmus Group, lnc
Schedule 1, Page 31 of '130
,ffi
Figure 7. PY 2013 Distribution of Estimated Annual UECs by Appliance Type
8888888886ots6oq..1.{6-diHd
Estimated Annual UEC (kwh)
.F|@ rRqfrirEltG
Table 18 presents the estimated, per-unit, average annual energy consumption for refrigerators and
freezers recycled by Avista in PY 2013. After the table, we describe how we adjusted these estimates to
arrive at gross per-unit saving estimates for participant refrigerators and freezers.
Table 19 presents the PY 2013 UEC results for Avista, compared to other utilities located in Canada and
the U.S. Avista's UECs are similar to the utilities we benchmarked.
E5 rsroeIEL rose4
oooooqq8-R8-
dtidd
Table 18. Estimate of Per-Unit Annual Energy Consumption
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 'l , Page 32 of 130
Table 19. Benchmarking: Average UEC Values
Avlna (lD, PY 20131
Avista (w4 PY 2012 & PY 2013)
Avista (lD, PY 2012)
Avista (wA & lD, PY 2011)
Avista (wA & tD, PY 2010)
Ontario Power Authority (2012)
Ontario Power Authority (2011)
Pacific Power (WA, 2O1l-2O72}
Rocky Mountain Power (lD, 2011-2012)
Rocky Mountain Power (UT, 20U-2012)
Rocky Mountain Power (WY, 2011-2012)
8
8i
6
5
6
5
8
8
1o
L
L,234
1,225
1,199
1,t47
1,158
1,153
L,240
L,239
L,2t7
L,323
7,2s6
1,139
1,098
1,177
1,O74
r,o73
L,270
t,L?2
L,O87
r,LIL
1,082
1.098
Part-Use
Part-use is as an adjustment factor specific to appliance recycling, which is used to convert the UEC into
average per-unit gross savings value. The UEC does not equal gross savings value, due to the following:
. The UEC model yields an estimate of annual consumption.
. Not all recycled refrigerators would have operated year-round ifthey had not been
decommissioned through the program.
The first time Cadmus used the UMP part-use methodology to evaluate an Avista program was in ldaho
for PY 2Ot2. Cadmus applied this methodology again for the PY 2013 ldaho evaluation.
While the UMP part-use methodology uses information from surveyed customers regarding pre-
program usage patterns, the final part-use estimate reflects the way appliances would likely be operated
if they had not been recycled (not how they were previously operated). For example, a primary
refrigerator operated year-round could become a secondary appliance and be operated part-time.
The UMP methodology Cadmus employed forthe PY 2013 evaluation accounts for potential shifts in
usage types. Specifically, we calculated part-use using a weighted average of the following, prospective
part-use categories and factors:
. Appliances that would have run fulFtime (part-use = 1.0).
. Appliances that would not have run at all (part-use = 0.0).
. Appliances that would have operated for a portion ofthe year (part-use between 0.0 and 1.0).
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
Khawaja, The Cadmus Group, lnc
Schedule 1, Page 33 of 130
S.
Using information gathered through the participant surveys,u Cadmus used the multistep process
outlined in the text below to determine part-use, as outlined in the UMP,
First, we used the survey information to determine if recycled refrigerators were primary or secondary
units (considering all stand-alone freezers as secondary units).
We asked participants who recycled a secondary refrigerator or freezer if the unit was unplugged,
operated year-round, or operated for a portion of the preceding year (assuming all primary units
operated year-round).
Cadmus asked participants who indicated that their secondary refrigerator or freezer operated for only a
portion of the preceding year to estimate how many months during that time their appliance was
plugged in. This subset of participants estimated 5.88 and 3,24 months for secondary refrigerators and
freezers, respectively. Dividing both values by 12 provided the annual part-use factors of 0.49 for all
secondary refrigerators and 0.27 for all freezers operated for only a portion of the year (Table 20).
rseloldaryunlsg4y , n=29
Not in Use A% O.OO
Used PartTime f3% 0.49 510
Used FullTime 79% 1.(x) f,238
weilhted avense i tm96 i o.ae 1,060
Att Units liamary n=4t n=t5 Iand Secondaryl
Not in Use 4% i O.OO - L2% i OOO -Not in Use 4% i 0.q) - L2% 0.00lrylj1ttl,r - q1- o:4s
--
6::!!:0.! ,2f, ]!l' 311 l
usea rrriii* lex i l.oo 1,238' 7s%- 1oo r,iss-fq4Ellft. --___r*l-g _ r*
Cadmus then asked participants how they would likely have operated their appliance if they had not
recycled it through the program. For example, if surveyed participants indicated they would have kept a
primary refrigerator independent ofthe program, we asked ifthey would have continued to use the
appliance as their primary refrigerator or would have relocated it and used as a secondary refrigerator.
We did not ask similar questions of participants who indicated they would have discarded their
12 Due to the relatively small number of ldaho participant survey respondents, cadmus combined the participant
suruey data from the Washington and ldaho PY 2013 surueys for the NTG analysis.
28
S.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
Khawaja, The Cadmus Group, lnc
Schedule 1, Page 34 of 130
Table 20. Historical Part-Use Factors by Category
appliance independent of the pro8ram, as the future usage of their appliance would be determined by
another customer.
Combining the historically based, part-use factors shown in Table 20 with participants' self-reported
action had the program not been available resulted in the distribution of likely future usage scenarios
and corresponding pert-use estimates. Table 21 shows the weighted average of these future scenarios,
revealing the program part-use factor for refrigerators (0.89) and freezers (0.78).13
Primary
Secondary
1.00
0.86
0.91
0.86
0.91
0.89
Kept (as primary unit)
rept (as iecondaryuniii
Discarded
Discarded
0.78 56%
44%
too%
29%
ro/J%
0.78
0.78
Table 22 presents the part-use factors compared with other utilities located in Gnada and the U.S.
Cadmus found that Avista ldaho has a similar part-use factor for refrigerators, and a slightly lower part-
use factor for freezers, than other utilities. The refrigerator part-use factor for PY 2013 is lower than for
PY 2072, but the freezer part-use factor is higher.
As the future usage type of discarded refri8erators cannot be known, Cadmus applied the weighted part-use
average of all units (0.89) to all refrigerators that would have been discarded !ndependent of the program.
This approach allows for disarded appliances to be used as primary or secondary units in a would-be
recipienfs home.
29
S.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
Khawaja, The Cadmus Group, lnc
Schedule 1, Page 35 of 130
Table 21. Part-Use Factors by Appliance Type
Table 22. Benchmarking: Part-Use Factors by Appliance Type
Avista 0D, PY 20131
Avista (tD, PY 2b12)
Avista (wA, PY ,012 & PY 2013)
Avista (wA & lD, PY 2O1O & PY 2011)
Ameren lllinois
Pacific Gas & Electric (2012)
Pacific Power (WA, 211.l-211.2l
Rocky Mountain Power (lD, 2011-2012)
Rocky Mountain Power (UT, 2011-2012)
Southern California Edison (2012)
8
7
8
6
5
1o
8
8
1o
L2
0.89
0.95
0.89
0.94
0.88
0.94
0.93
0.84
0.93
0.94
0.78
o.74
7.ez
0.82
ose
0.90
0.93
o.9o
Net-to-Gross
Cadmus used the following formula to estimate net savings for recycled refrigerators:
Net Sauings = Gross Savings - Freeridership and. Second.ary Matket Impacts
- lnduced Replacement
Gross savings are the evaluated in situ UEC for the recycled unit, adjusted for part-use. Freeridership and
secondary market impacts are program savings that would have occurred in the program's absence.
lnduced replacement is the average, additional energy consumed by replacement units purchased due
to the program.
Applying the UMP protocol introduced an additional parameter related to net savings-secondary
market impacts-which required Cadmus to use a decision-tree approach to calculate and present net
program savings. This decision tree-populated by the responses of surveyed participants-presented
savings under all possible scenarios of what could happen to the discarded equipment. Cadmus used a
weighted average of these scenarios to calculate net savinSs attributable to the program. The text below
includes specific portions ofthe decision tree to highlight specific aspects ofthe net savings analysis.
Freeridership
To determine freeridership, Cadmus first asked participants if they considered discarding the
participating appliance prior to learning about the program. lf the participant did not indicate a previous
consideration to dispose ofthe appliance, Cadmus categorized them as a non-freerider and excluded
them from the subsequent freeridership analysis.
30
S.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
Khawaja, The Cadmus Group, lnc
Schedule 1, Page 36 of 130
Next, Cadmus asked all remaining participants (i.e., those who had considered discarding their existing
appliance before learning about the program) a series of questions to d'etermine the distribution of
participating units likely to have been kept versus those likely to have been discarded absent the
program. Three scenarios independent of program intervention could have occurred:
. The unit would be discarded and transferred to someone else.
o The unit would be discarded and destroyed.
. The unit would be kept in the home.
To determine the percentage of participants in each of the three scenarios, Cadmus asked surveyed
participants about the likely fate of their recycled appliance had it not been decommissioned through
the program. Cadmus categorized their responses into the following options:
. Kept the appliance.
. Sold the appliance to a private party (either an acquaintance or through a posted
advertisement).
o Sold or gave the appliance to a used appliance dealer.
. Gave the appliance to a private party, such as a friend or neighbor.
. Gave the appliance to a charity organization, such as Goodwill lndustries or a church.
o Had the appliance removed by the dealer who provided the new or replacement unit.
. Hauled the appliance to a landfill or recycling center, or had someone else pick it up forjunking
or dumping.
Cadmus also asked surveyed participants if they had considered getting rid of their old appliance before
hearing about the program. The distribution oftheir responses to this question are summarized in Table
23.
Yes
No
iotrt
Once Cadmus determined the final assessments of participants' actions independent of the Second
Refrigerator and Freezer Recycling Program, we calculated the percentage of refrigerators and freezers
that would have been kept or discarded (Table 24).
31
Table 23. Distribution of Participants' Pre-Program Disposal lntentions
s.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
Khawaja, The Cadmus Group, lnc
Schedule 'l , Page 37 of 130
Table 24. Final Distribution of Kept and Discarded Appliance
!9?l
Discarded
Total
L8%
varies by Discard Method
1m96 1(x,%
Cadmus benchmarked these values against Avista ldaho's PY 2012 evaluation and those of other
appliance recycling programs in ldaho, Utah, Washington, and Wyoming, as shown in Table 25. Avista's
PY 2013 result for ldaho is most similar to Rocky Mountain Powe/s Wyoming result, though the
percentage of freezers likely to be kept is higher than any of the benchmarked programs. Within the
Avista ldaho program, the percentage of refrigerators likely to have been kept decreased relative to PY
2012, though the percentage for freezers increased substantially,
No
Avlsta 0D, PY 20131
lvista'(to, py zoii)
evista 1w4 ev zorz a ev zoral
Pacific Power (WA, 2OL]-zO].zl
Rocky Mountain Power (lD, 20i1-2012)
Rocky Mountain Power (UT, 2O1l-2012)
Rocky Mountain power (WY, 2O:-]-2OLzl
7
8
8
25%
1816
36%
22%
3L%
22%
a 32% 29%
10 2W 24%
4 L5% 27%
S ec o ndary M orket Impocts
lf. absent the program, a participant would have directly or indirectly (through a market actor)
transferred the program-rerycled unit to another Avista customer, Cadmus determined what actions the
would-be acquirer might have taken with that unit.
Some would-be acquirers would find another unit; others would not. This reflects that some acquirers
would be in the market for a refrigerator (and would acquire another unit), while others would not (and
would have taken the unit opportunistically). Absent program-specific information, it is difficult to
quantify chanSes in the total number of refrigerators and freezers in use (overall and specific to used
appliances) before and after implementing the program. Without this information, the UMP
recommends assuming that one-half of the would-be acquirers would obtain an alternate unit. Without
information to the contrary, Cadmus applied the UMP recommendation to this evaluation.
32
Table 25. Benchmarking Kept Appliances
Exhibit 3
Case Nos. AVU-E-14 AVU-G-'!4
S. Khawaja, The Cadmus Group, lnc
Schedule 1, Page 38 of 130
Next, Cadmus determined what percentage ofthe alternate units would Iikely be another used
appliance (similar to those recycled through the program) versus a new, standard-efficiency unit
(presuming fewer used appliances remained available due to program activity).14
As discussed, estimatinB this distribution definitively proves difficult. The UMP recommends taking a
midpolnt approach when primary research is unavailable: evaluators should assume that one-half of the
would-be acquirers would obtain a similar used appliance, and one-half would acquire a new, standard-
efficiency unit.
Cadmus used the ENERGY STAR websiter5 to determine the energy consumption of new, standard-
efficiency appliances. Specifically, Cadmus averaged the reported energy consumption of new, standard-
efficiency appliances of comparable sizes and configurations as the program units.
Figure 8 details Cadmus' methodology for assessing the program impact on the secondary refrigerator
market and for applying the recommended midpoint assumptions when primary data were unavailable.
As shown, accounting for market effects resulted in three savings scenarios:
o Full per-unit gross savings;
o No savings; and
. Partial savings (i.e., the difference in energy consumption between the program unit and the
new, standard-efficiency appliance that was acquired instead).
Figure 8. Secondary Market lmpacts-Refrigerators
Integration of Freeridership and Secondary Market Impacts
After estimating the parameters of the freeridership and secondary market impacts, Cadmus used the
UMP decision tree to calculate the average, per-unit program savings, net of their combined effect.
Figure 9 shows how Cadmus integrated these values into an estimate of savings, net of freeridership and
secondary market impacts. Cadmus calculated the weighted average freeridership and secondary
The would-be acquirers could also select a new ENERGY STAR unit. However, Cadmus assumed that most
customers in the market for a used appliance would upgrade to the next lowest price point (a standard-
efficiency unit).
http://ww.enerqvstar.sov/index.cf m ?f useaction=ref ris.calculator.
i3
S.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
Khawaja, The Cadmus Group, lnc
Schedule 1 , Page 39 of 1 30
market impacts (778 kwh per unit) as the sum product of the program proportions and the per-unit
enerSy consumption with the program for each scenario.
Figure 9. Savings Net of Freeridership and Secondary Market lmpacts-Refrigerators
I r,r0r t_l r.rff l-r " )tryjtsqN$lj - t:{rElt!!Er9j - \--:J
- G.*f,**J = Ga
- a--------l = @
- a-;----] = @
(-----l---__] = @
N5-f i-gl.rh: Fffiip d !@ .el rllB CD
Induced Replacement
The UMP states that evaluators must account for the energy consumption of replacement units onry
when the program induced that replacement (i.e., when the participant would not have purchased the
replacement refrigerator without the recycling program),
ln the case of non-induced replacements, the energy consumption of the replacement appliance does
not prove germane to the savings analysis, as the appliance would have been purchased or acquired
regardless of the program. The acquisition of another appliance in conjunction with participation in the
program does not necessarily indicate induced replacement. ABain, this is consistent with the methods
outlined in the uMP.
Cadmus used the results of the participant surveys to determine which replacement refrigerators and
freezers program participants acquired due to the program. Survey results indicated that the program
reduced the total number of used appliances operating within Avista's ldaho service territory, and that
the program raised the average efficiency of the active appliance stock.
Cadmus then used participant survey results to estimate the proportion of replacements induced by the
custome/s participation in the program. Specifically, Cadmus asked each participant that indicated they
replaced the participating appliance: "Would you have purchased the replacement appliance without
the 530 incentive you received for recycling the old one?"
As a 530 incentive will likely not provide sufficient motivation for most participants to purchase an
otherwise unplanned for replacement unit (which can cost 5500 to 52,000), Cadmus asked a follow-up
question of participants who responded "No." lntended to confirm the participant's assertion that only
34
S.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
Khawaja, The Cadmus Group, lnc
Schedule 1 , Page 40 of 1 30
the program caused them to replace their appliance, the question was: "Just to confirm: you would not
have replaced your old refrigeratorfreezer without the Avista incentive for recycling, is that correct?"
To further increase the reliability ofthese self-reported actions in the induced replacement analysis, we
also considered whether the refrigerator was the primary unit and the participant's stated intentions in
the program's absence.
For example, if a participant would have discarded their primary refrigerator independent of the
program, the replacement could not be program induced (since it is extremely unlikely a participant
would live without a primary refrigerator). However, for all other usage types and stated intention
combinations, induced replacement was a viable response.
As expected, results indicated that the program only induced a portion of the total replacements: the
program induced 8% of all refrigerator participants and 8% of freezer participants to acquire a
replacement unit, as shown in Table 25.
Refrigerator
Freezer
As shown in Table 27, Avista's induced replacement was higherthan both the comparison utilities and
higher than Avista's previous evaluations, and was most similar to Rocky Mountain Powe/s 2011-2012
results in ldaho.
a%
a%
Avista (lo, PY 2013)
Avista 0D, PY 2012)7
6
8
8
8
10
4
8%
o%
4%
tt%
5%
7%
4%
5%
a%
o%
4%
7%
4%
7%
3%
2%
Avista (wA & tD, PY 2010 & PY 2011)
lvista 1wa, ev zor2 & PY 2013)
Pacific Power (WA, 2OLL-zOl2l
Rocky Mountain Power (lD, 2011-2012)
Rocky Mountain Power (UT, 2011-2oiij
Rocky Mountain Power (WY, zott-iotz1
Figure 10 shows Cadmus'calculated induced replacement within the decision tree. Cadmus calculated
the weighted average induced consumption per unit as the sum product ofthe program proportions and
the per-unit energy consumption resulting from the program.
35
Table 25. lnduced Replacement Rates
Table 27. Benchmarking: lnduced Replacement
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
Khawaja, The Cadmus Group, lnc
Schedule 1, Page 41 of 130
S.
Figure 10. lnduced Replacement Refrigerators
m@ m 6ffifi mffirww tre mam- mxMfi
=@ffiu$sl&wcC;,*";l-G;*;=cD
Final NTG
As summarized in Table 28, Cadmus determined final net savings as gross savings and spillover savings,
minus freeridership, secondary market impacts, and induced replacement,
Summary ot lmpact Flndings
Using the above per-unit values, Cadmus calculated the total program savings for the PY 2013 Second
Refrigerator and Freezer Recycling Program in ldaho as 117,699 kWh, after adjustments (as shown in
Table 29).
Table 29. ldaho PY 2013 Second Refrigerator and Freezer Recycling Program Savings
Refrigerator Recycling
Freezer Recycling
I rotar
-As shown in Table 30, Avista's NTG for refrigerators is less than all the benchmarked programs. This NTG
result was driven downward from the previous evaluation, primarily due to the ratio of appliances that
would have been discarded absent the program, as well as to the mature nature of the program relative
to other programs. The NTG for freezers, however, is similar to the benchmarked programs.
Table 28. PY 2013 NTG Ratios (kwh)
Exhibit 3
Case Nos. AVU-E-14 AVU-G-'|4
S. Khawaja, The Cadmus Group, lnc
Schedule 1, Page 42ot 130
Table 30 Benchmarking NTG Ratios
Avista (lD, PY 2013)
Avista (lD, PY 2012)
ariit" 1wa, PY 2012 & PY 2o1j)
Avista(wA & tD, PY 2010 & PY 2011)
Oniario Power Authority (2012)
Ontario Power Authority (2011)
Pacific Power (CA, 2009-2010)
Pacific Power (WA, 2OLL-2O721
Rocky Mountain Power (lD, 2011-2012)
Rocky Mountain power (ur, zoir-zo1i)
Rocky Mountain Power (WY, 2077-2oL2l
1.3.4. ENERGY STAR Products
8
7
8
6
6
5
3
8
8
1o
4
28%
46%
4L%
57%
47%
53%
64%
51%
54%
56%
39%
52%
33%
sa%
s6%
4a%
53%
67%
5L%
4S%
isx
s1%
fuogram Description
The ENERGY STAR Products Program includes the following measures:
o Clothes Washer (Electric and Gas)
o Freezer (Electric)
. Refrigerator(Electric)
Through the program, Avista offers direct financial incentives to motivate customers to use more
energy-efficient appliances; this indirectly encourages market transformation by increasing the demand
for ENERGY STAR products. The program includes electric and gas measures, but Cadmus only considers
electric savings in this report.
Analysis
Energy savings credited to the ENERGY STAR Products Program had to meet the following criteria:
. Measures had to remain in place and be operating properly at the time of verification;
. Numbers of installed equipment pieces and their corresponding model numbers in the
applications had to match the database; and
. Units must have been ENERGY STAR-qualified at the time of the program offering.
37
S.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
Khawaja, The Cadmus Group, lnc
Schedule 1, Page 43 of 130
Clothes Washerc, RefriSerators, and Freezers
Energy-saving calculations drew upon a 2009 Cadmus study,16 which metered more than 100 clothes
washers in California homes for three weeks-the largest ,n situ metering study on residential clothes
washers and dryers conducted in the last decade. Cadmus has updated the analysis since the 2012
Avista TRM was completed to improve the accuracy of the savings estimated.
Dryers produced the majority of energy consumption and savings, as high-efficiency washing machines
removed more moisture from clothes, allowing for shorter drying times.
Determining adjusted gross savings required using the following, additional input assumptions:
o Based on recent independent evaluation surveys from the RBSA17 and on PY 2012 Avista
participant surveys, Cadmus estimated 252 washing cycles per year. We adjusted the UES values
accordingly, which is reflected in this measure's realization rate.
o Cadmus used data from the California metering studyto estimate consumption perwash and
dry cycle for the base and efficient equipment.
Results ond Findings
Table 31 shows total reported and qualified counts, savings, and realization rates for electric ENERGY
STAR Products Program measures in ldaho.
21,479
294 too%100%
Electric
Refrigerator 110 4,840
325 l
l
7,207 1000/6 100%
326
7,207
Ltt%
149%
Prcfiami :'-':-" 27, 7gl;t4: 29,011 i L@% too,| zst,aiI Total i l
Progrdm Totdl sdvings mdy be dillerent that the sum olthe vdlues shown due to rounding.
t7
37x.
The cadmus Group, lnc. "Do the Savings Come Out in the Wash? A Large Scale Study of ln-situ Residential
Laundry Systems." 2010. Available online:
htto://M.aceee.orrlfi les/proceed inss/2010/data/oaoers/2223.odf
Ecotope lnc. 2O1, Residentiol Building Stock Assessment: Single-Fomily Chdtdcteristics and Eneryy U*. Seallle,
wA: Northwest Energy Efficiency Alliance. 2012.
38
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule '1, Page 44 of 130
Table 31. PY 2013 ENERGY STAR Products Program Results in ldaho
The program achieved a 37% realization rate. Our review of program applications determined that 42%
ofthe applications with water heater fuel originally marked as "Natural Gas" had been processed as
"Electric." Cadmus reviewed gas consumption data from the associated premises to determine if these
customers have electric service only or if they have both electric and natural gas service, The adjusted
savings for this measure accounts for the existence of gas hot water heaters for 42% of the units
rebated. These units still deliver electricity savings since the majority of homes are assumed to use
electric dryers.
1.3.5. Heating and Cooling Efficiency
Progrom Description
The Heating and Cooling Efficiency Program included the following electric equipment:
. Ductless Heat Pump (DHP)
. Air-source Heat Pumps (ASHP)
o Variable Speed Furnace Fan
Analysis
The PY 2010 and PY 2011 electric impact evaluation reportlE documented the analysis Cadmus
performed to determine the change in energy consumption resulting from the installation ofelectric
heating and cooling measures. As that analysis continues to provide the best information on these
measures, Cadmus retained those results for PY 2013.
Results and Findings
Table 32 shows total tracked and qualified counts, savings, and realization rates for electric Heating and
Cooling Efficiency Program measures in ldaho. The program achieved a 94% realized adjusted gross
savings rate. The reduction in savings is due to differences between the values used to track savings for
the program and the savings shown in the 2012 Avista TRM.
18 Cadmus. Avista 2010-2017 Multi-sector Electric lmpoct Evoluotion Report, May 2012
39
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 1, Page 45 of 130
Table 32. Heating and Cooling Efficiency Program Results*
*Table values may not sum due to rounding.
1.3.6. Space and Water Conversions
Ptogram Description
Through the Space and Water Conversions Program, Avista incents three measures for residential
electric customers who currently use electricity to heat the space and water in their homes, but have
the opportunity to use natural gas or switch to an ahernative, more efficient technology that uses the
same fuel source. The equipment conversions during PY 2010 through PY 2013 included the following
measures:
. El€ctric Forced Air Furnace to Air Source Heat Pumps (ASHP)
. Electric Forced Air Fumace to Natural Gas Forced Air Furnace (NGF)
. Electric Water Heaterto Natural Gas Water Heater (NGWH)
By offering conversion rebates, Avista seeks to achieve energy efficiency by changing the fuel mix used
by customers, which leads to savings from the lower-priced fuel (in case of a conversion from an electric
furnace to a NGF and electric water heater to a NGWH) and to higher efficiency in overall cooling and
heating usage.
With the residential energy-efficiency programs, Avista targets single-family homes and units in
multifamily buildings. Avista customers started participating in the conversion rebates in PY 2010. Table
33 shows participation by conversion measure and year, in both ldaho and Washington. Avista phased
out conversion rebates in ldaho in PY 2013 for conversion from an electric water heater to a NGWH.
Table 33 shows the number of participant that installed any ofthe conversion measures, grouped by
year of installation.
40
S.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
Khawaja, The Cadmus Group, lnc
Schedule '1, Page 46 of 130
@
Table
PY 2010
I PY 2011 51 74 :ASHP' PY20L2 60 il i
ii PY 2013 48 55 r
pv ioro s1 82
PY 2011 27 55NGF :PY20L2 24 74
PY2o13 2A 78
I PY2010 22 9s
PY 2011 16 79NGWH iI PY 2Ol2 15 75 :
:lI10L3_ _ s _1s:* This column double-counts participants who installed multiple measures.
* This primarily consists of all customers who installed a NGF and NGWH.
I mpact Evo I uotion Methodology
With the impact evaluation, Cadmus sought to estimate the change in energy use after installing these
conversion measures. More specifically, Cadmus' evaluation of the Space and Water Conversions
Program consisted of the following three tasks:
1. Data collection, review, and preparation.
2. Billing analysis.
3. Energy-savingsestimations.
Data Collection, Review, and Preparation
To perform the billing and uplift analysis, Cadmus collected the data outlined below.
Monthly Customer Bills
Cadmus collected data about monthly gas and electricity bills between January 2010 and December
2013. The data included approximately 10 to 12 months of bills prior to the measures installations and
the same number of months of bills after the installations. These billing data included: account numbers,
energy use during the monthly billing cycle, and the last day of the billing cycle. Avista supplied these
data to cadmus.
33. Participation in Fuel Conversion Program by Year and State
L23 !29
135
124
113
133
42992
98
105
Lt7
95
9o
50
Table 34. Number of Homes That Participated From PY 2010 Through PY 2013
41
S.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
Khawaja, The Cadmus Group, lnc
Schedule 1, Page 47 oI 130
Progrom lnformation
Cadmus obtained measures data from Avista. These data included: program tracking data for the PY
2011-PY 2013 participants, account numbers and site lDs for linking to billing data, all the measures
installed, rebated amounts of therms and kWh saved, and application dates for the rebates.
Weother
Cadmus collected National Climatic Data Center daily average temperature data from 2010 through
January 2014 for eight weather stations: two in ldaho (Lewiston and Coeur D'Alene) and six in
Washington (Moses Lake Grant Co., Walla Walla, Spokane, Fairchild, Felts, and Pullman Moscow). These
were the stations nearest to all the program homes in the Avista territory.
Data Preparation
Cadmus prepared billing data for analysis using the following steps:
. Reformatting and merging the raw billing data for all customers.
. Separating the gas and electricity datasets and identifying customers that had dual usage
(electricity and gas) versus customers using only electricity.
. Renaming the market measure description, such as the following the same conversion measure
naming convention for all program years.
. ldentifying homes that had multiple conversions and assigning them to a separate group.
. Specifying the pre- and post-periods for each customer account:
I The Customer'pecific Meosure lnstall Date: For each custome/s unique installation date,
this specification compares the year ending just before the install date with the year
beginning on the installation month.
. The Full Yeor: ln this specification, the install year is taken as the current year and the
energy consumption of the full year before the current year is compared to the full year
after the current year,
Table 35 shows an example ofthe specification of the pre- and post-installation periods under the two
specifications. ln this analysis, Cadmus has used a combination of the two specifications. While the
Customer- Specific Measure lnstall Date specification allows the data from a more compressed
timeframe to be used, it relies heavily on the exact installation date. The Full Year specification excludes
this uncertainty by assuming that the conversion installations occurred any time during the rebate
application year. The Full Year specification requires at least three years of data, ln cases where this
requirement was not met, Cadmus used the Customer-Specific Measure lnstall Date specification.
42
S
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
Khawaja, The Cadmus Group, lnc
Schedule 1, Page 48 of '130
@-
Table 35. Example of Pre- and Post-lnstallation Period Under
Customer Specific Measure lnstall Date
Full Year
the Two Specifications
June 2010
Cadmus used daily average temperature and billing cycle information to estimate cooling degree days
(CDDs) and heating degree days (HDDs) for each home during the billing cycle. This required using a base
temperature of 55 degrees and billing cycle end dates to calculate HDDs and CDDS that exactly matched
days in the custome/s bill.
Based on the conversion group (electric furnace to NGF only, electric water heater to NGWH only, both
electric furnace to NGF and electric water heater to NGWH, and ASHP) and the fuel usage type (electric
only and dual fuel: electric and gas), Cadmus estimated six separate models. The next section outlines
the selected sample sizes in these six groups.
Data Attrition
Cadmus performed billing analysis on the population of program homes, except for homes from the
estimation sample that satisfied one or more of the following criteria:
o The home had fewer than 11 pre- or post-program monthly energy bills.
The home did not pass PRISM modeling screens, which are based on the weather-normalized pre- and
pre- and post-installation annual usage. These are discussed in more detail in the
o Billing Analysis section.
Table 35 shows the total customer accounts that had a conversion measure and the final sample
Cadmus used in the PRISM and the regression analyses. Each row in the table indicates the accounts
remaining after attrition.
June 2009 to May 2010 June 2010 to April 2011
January 2009 to January 2011 to December
December2oog 2Ol7
43
S.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-'|4
Khawaja, The Cadmus Group, lnc
Schedule 1 , Page 49 of '130
Table 36. Sample Size Selection for PRISM Analysis
623 1,361measures
r-ow usageleEtt* r,om iwfrf irl
pre- or post-installation period 52 l 512
I
1,30950346 i 301
47 203 2s
199 I 2s
840
82:
667
4916
post-installation period and/or 2) usage that increased by more than 83% between the pre- and post-lnstallation period.
t* The numbers in bold are the final sample size Cadmus used for the per-home savings estimation.
Bllllng Analysis
To estimate program electricity savings, Cadmus used two approaches: PRISM and fixed-effects
regression. Cadmus first estimated the PRISM model to obtain weather-normalized annual consumption
(NAC) and identify outliers. Cadmus then estimated a regression model to control for the installation of
other weatherization measures or efficient equipment. Details on the model specifications can be found
in Appendix A.
Progrum lmpact Evoluotion Findings
Per Home Savings tmpacts (PRISM)
Table 37 summarizes the PRISM resuhs for conversion measures across the six groups. The results show
the annual savings, relative precision on these savings, the pre-NAC for each group, and the savings as a
percentage ofthe pre-NAC. Table 37 also reports savings as a percentage ofthe pre-conversion period
heating load.
tu
S.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
Khawaja, The Cadmus Group, lnc
Schedule 1, Page 50 of 130
Total accounts wlth sufficient
billing data for PRISM analysis
Table 37. Electric Savings per Home (PRISM Results)
NGF Dual
NGWH Dual
Multiple Dual
Electric
OnlvASHP
Dual
All Homes
8%
L3%
rg%
10%
38%
10%
164
159
23
288
33
32r
9,563
4,367
123so
4,419
4,994
4,478
24,349
15,305
25,646
24,955
24,566
24,9t5
7t%
97%
9L%
39% 13,433
27% 4,505
48% 13,558
t8% 15,181
20% L2,944
t8% 14,951
29%
39%
30%
The evaluated savings for electric furnace to NGF conversion resulted in annual savings of 9,553 kwh
per home (39% of pre-conversion usage and 71% of pre-conversion heating usage) with a relative
precision of t8%. For electric water heater to NGWH conversions, the annual savings are 4,357 kwh per
home l27Yo of pre-conversion usage and 97% of pre-conversion heating usa8e) with a relative precision
of t13%. The homes with both furnace and water heater conversions had on average 12,350 kWh of
savings (48% of pre-conversion usage and 91% of pre-conversion heating usage) with a relative precision
ot tt9%.
The following figures are based on PRISM model results. Figure 11 shows the distribution of percentage
changes in the predicted electricity use between the pre- and post-conversion periods.
Figure 11. Distribution of Percentage Changes in Annual Electricity Savings by Conversion Group
100
Ero
EroEo40
ooE20az
0
IASHP TNGF NGWH
-0.3 -o.2 -0.1 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
These results show an approximate normal distribution centered around a 30% reduction in electric use
for ASHP conversions, 50% reduction for NGF conversions, and 35% for NGWH conversions.
Figure 12 shows the distribution of percentage changes in the predicted electricity use for heating
between the pre- and post-conversion periods. The percentage changes are based on the pre-period
heating load.
4S
Exhibit 3
Case Nos. AVU-E-14 AVU-G-'|4
S. Khawaja, The Cadmus Group, lnc
Schedule 1, Page 51 of 130
ffi
for Heating
50
Es0cP+oEos30tb205
5roz0
Figure 12. Distribution of Percenta8e Changes in Annual Electricity Use
-0.3 -0.2 -0.1 0.0 0.1 0.2 0.3
IASHP INGF
0.4 0.5
NGWH
1.00.90.8o.70.6
The figure shows a more than 80% drop in the heating load for approximately 70% of electric furnace to
NGF conversion homes. For the electric water heater to NGWH conversion homes, there is varying
amounts of heat load savings across all homes. Almost 50% of savings were achieved for most ASHP
conversion homes.
Figure 13 shows the distribution of percentage changes in the predicted electricity use for cooling
between the pre- and post-conversion periods. The percentage changes are based on the pre-period
cooling load.
Figure 13. Distribution of Percentage Changes in Annual Electricity Use for Cooling
The figure shows that customers achieved cooling efficiency, especially with ASHP conversions, followed
by NGF conversions, then NGWH conversions.
Per Home Savings lmpacts (Pooled Regression Model)
Cadmus ran several specification of the panel regression model. We found that the overall savings
results were fairly consistent across the PRISM and pooled regression model. ln the final model, Cadmus
controlled for all additional non-program measures installed by the conversion participants (except for
46
35
E30G'3 zs
H20ctls
-t 10E-!5z 0
IASHP TNGF NGWH
-0.3 -0.2 -0.1 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
Khawaja, The Cadmus Group, lnc
Schedule 1, Page 52 of 1 30
S.
high-efficiency variable speed motors). The results for this model are shown in Table 38. Cadmus used
the coefficient estimates and standard errors from this table to calculate the savings and relative
precision.
NGF Dual
NGWH Dual
Multiple Dual
Electric Only
ASHP Dual
Ail
L64 ]..0,287 9%
159 4,370 t6%
23 13,643 25%
288 4,775 rr%
33 5,309 30%
321 e,gie 10%
24,349
16,30!
25,646
24,955
zi,isa
24,91,5
The results reveal that there are higher savings for each conversion group after controlling for the
installation of other measures.
Table 39 provides the percentage of conversion participants in each group who had additional/non-
program measures installed. The regression savings analysis controls for all additional measures except
high-efficiency variable speed motors.
Table 39. Percentage of Additional Measures lnstalled by the Conversion Participants
NGF
NGWH
nsnp
27%
26%
45%
33%
27%52%
Results ond Findings
Table 40 shows the total tracked and qualified counts, savings, and realization rates for electric Space
and Water Conversions Program measures in ldaho.
47
S.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
Khawaja, The Cadmus Group, lnc
Schedule 1, Page 53 of 130
Table 38. Electric Savings per Home (Fixed-Effects Model)
E Electric to 4A 312,492ASHP
ryI
Table 40. Space and Water conversions Measures and Reported and Adjusted Savings
334,536 ?67,7il
6 21,636 26,202
2L2,tL2
ProSram
fotal g2 568,664 505,078
1.3.7. Weatherization/Shell
Progrum Description
Avista offered the Weatherization/Shell Program, for which it incented three measures available to
residential customers who heat their homes with fuel provided by Avista:
. lnsulation-ceiling/Attic
. lnsulation-Floor
o lnsulation-Wall
Avista incented qualifring ceiling and attic insulation (both fitted/batt and blown-in) that increased the
R-value by 10 or more at 50.15 per square foot. Homes qualified if they had existing attic insulation of R-
19 or less.
Avista incented floor and wall insulation (both fitted/batt and blown-in) that increased the R-value by 10
or more at 50.20 per square foot. Homes were eligible if they had existing floor and/or wall insulation of
R-5 or less.
Analysis
Cadmus conducted a statistical billing analysis to determine adjusted gross savin8s and realization rates
for installed electric weatherization in PY 20U, PY 2012, and PY 2013. The previous years' billing
analyses primarily included PY 2010 customers, although we extrapolated the realization rates to PY
20U. We included PY 2011customers in the PY 2013 billing analysis since they now have complete post-
period billing data. This increased the sample sizes and improved the precision of the weatherization
savings estimates.
We also present results that only include PY 2012 and PY 2013. To increase the accuracy of our analysis,
we only included participants with at least 10 months of pre- and post-installation billing data.
Consequently, the PY 2013 billing analysis includes PY 2OLL,N 2012, and early PY 2013 participants.
tt8
Exhibit 3
Case Nos. AVU-E-14 AVU-G-I4
S. Khawaja, The Cadmus Group, lnc
Schedule 1, Page 54 of 130
To estimate weatherization energy savings resulting from the ldaho program, Cadmus used a pre- and
post-installation combined CSA and PRISM approach, We calculated overall electric model savings
estimates for each measure bundle. We also attempted to estimate the detailed measure-specific
savings impacts.
Billing Analysis Methodology
Avista provided Cadmus with monthly electric billing data for all ldaho participants, from January 2009
through January 2014. Avista also provided a measure detail file containing participation and measure
data. Participant information included:
. Customer details,
. Account numbers,
. Types of measures installed,
. Rebate amounts,
. Measure installation costs,
. Measure installation dates, and
. Deemed savings per measure.
Cadmus first matched weatherization measure information with the electricity billing data. We obtained
daily average temperature weather data from January 2009 through January 2014 for National Oceanic
and Atmospheric Administration (NOAA) weather stations, representing all ZIP codes in Avista's service
territory. From daily temperatures, we determined base 55 HDDS and CDDS for each station. Using ZIP
code mapping for all U.S. weather stations, we determined the nearest station for each ZIP code. We
then matched billing data periods with the HDDs and CDDs from the associated stations.
Cadmus specified the pre- and post-installation periods for each customer account using two
specifications:
L. The Customer-Spectic Medsure lnstoll Ddte: For each customer's unique installation date, this
specification compares the year ending just before the install date with the year beginning on
the installation month.
2. The Fixed Dotes: For this specification, the earliest and latest dates of available billing data are
selected. ln effect, we used the period ofJanuary 2010 through December 2010 as the pre-
installation period, before any installations occurred. We defined the post-installation period as
the latest period with complete billing data: February 2013 through January 2014.
Table 41 shows an example ofthe specification ofthe pre- and post-installation periods underthe two
specifications. ln this analysis, Cadmus used a combination of the two pre-post specifications. While the
Customer-Specific Measure lnstall Date specification allows for data from a more-compressed
timeframe to be used, it relies heavily on the exact installation date. The Fixed Dates specification
removes this uncertainty by keeping only the earliest and latest periods of data, which are well outside
the installation period. The drawback with using Fixed Dates is that it requires a longer billing data
S.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
Khawaja, The Cadmus Group, lnc
Schedule 1, Page 55 of 130
history; however, Cadmus relied on this method by default. To minimize the attrition, we used the
Customer Specific Measure lnstall Date specification when there was insufficient billing data to use
Fixed Dates.
Table 41. Example of Pre- and Post-lnstallation Period Under the Two Specifications
customer-specific Measure lnstall Date November 2011 -
October 2012
November 2012 -
october 2013
Fixed Dates
November 2012
January 2010 -
December 2010
February 2013 -
January 2014
Data Screening
General Screens
Cadmus removed accounts with fewer than 10 paired months (300 days) of billing data in the pre- or
post-installation period, which could have skewed the weatherization savings estimates.
PRISM Modeling Steens
As a second step of the data screening process, Cadmus ran PRISM models for pre- and post-installation
billing data. These models provided weather-normalized pre- and post-installation annual usage for each
account, and provided an alternate check of the savings obtained from the CSA model. Details on the
model specifications can be found in Appendix A.
After running the three models, we dropped models with a negative heating and/or cooling slope. The
best of the remaining models for each customer in either the pre- or post-installation period had the
highest R-square with positive heating and cooling slopes.
Next we applied the following screens to the PRISM model output, removing outlier participants from
the billing analysis:
. Accounts wherc the postinstollotion weother-normalized usage wos 7096 hlgher ot lower than
the pre- NAC usoga Such large changes could indicate property vacancies or adding or
removing other electric equipment that is unrelated to weatherization (such as pools or spas).
. AccounFwkh negotive intercepts (fuse load).These negative intercepts indicate a negative
base load, such as for lighting, refrigerators, or plug loads. ln electric homes, the base load is
never expected to be negative.
. Accountswherethe prc-cnd post-instdllatlon billlngdota hod anomalies, includingvacancles,
seasonol usoge, outltec, ond/ot equlpment changes.
The ldaho weatherization population included 159 participants. Once we screened the data,66 ldaho
weatherization participants (39%) remained for use in the CSA model, outlined below, to determine
overall savings.
50
S.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
Khawaja, The Cadmus Group, lnc
Schedule 1, Page 56 of 130
Table 42 summarizes the attrition from each ofthe screening steps listed above. Each row in the table
indicates the accounts remaining after attrition. Approximately tl4% of the participant accounts were
dropped because they did not have sufficient pre- and post-period billing data in the analysis. Another
tTyowerc dropped from PRISM screening, and from the presence ofvacancies, seasonal usage, outliers,
or equipment changes in the billing data.
Total ldaho weatherization accounts 159
Matched to billing data provided 169
Less than 10 months of pre- or post- billing data 94
PRISM screening* 84
Vacancies, seasonal usage, outliers, and/or
equipment changes 66
100%
100%
55%
50%
39%
0
0
75
10
18
0%
o%
44%
6%
t7%
Flnal analysis group , 55 39% 103 6l%
* Using PRISM screens, Cadmus dropped accounts with: 1) negative heating slopes in the pre- or the post-period or
2) post"period usage that changed by more than 70% from pre-period usage.
CSA Modeling Approach
To estimete weatherization energy savings from this program, we used a pre/post CSA, fixed-effects
modeling method, using pooled monthly time-series (panel) billing data. This fixed-effects modeling
approach corrected for differences between pre- and post-installation weather conditions, as well as for
differences in usage between participants, through the inclusion ofa separate interceptforeach
participant. This modeling approach ensured that model savings estimates would not be skewed by
unusually high-usage or low-usage participants. Details on the model specifications can be found in
Appendix A.
Progrdm lmpoct Evoluotion Findings
Overall Savings lmpacts
Table 43 summarizes the usage and savings associated with the weatherization measures installed in
electrically heated homes in ldaho and Washington.le The results show the annual savings, relative
precision on these savings, the pre-installation heating usage NAC for each level, and the savings as a
percentage of the pre-heating usage NAC. The table also shows ex onte savings estimates and the
achieved realization rates for the weatherization measures.
rt Cadmus also estimated measure-level models for PY 2012 and PY 2013 that contain the most recent ex orte
estimates. These estimates revealed that the attic insulation model savings were generally higher than the
current ex orte values. The wall insulation model savings were similar to the ex ante savings, and the floor
insulation model savings were lower than the ex onte savings.
51
Table 42. Weatherization Account Attrition
S.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
Khawaja, The Cadmus Group, lnc
Schedule 1, Page 57 of 130
Table 43. ldaho and Washington Combined Weatherization Electric Savings per Home
(Fixed-Effects Model)
ldaho only sample
PY 2011-PY 2013 56
PY 2012-PY 2013* L4ll
PY 2011 52 ,.
20,813 11,125zi,tii i,i)io,sq' r,gi6
2,O29
t,540
2,368
35%
gay"
37%
Comblned Washington & ldaho Sample
PY 2011-PY 2013 2,25 2,3L5
PY 2011 2,24L 2096 19,145 10,620 tt.7%
Overall, the ldaho PY 20U-PY 2013 weatherization measures achieved savings of 2,020 kWh, or 9.7%
relative to the pre-installation period NAC. With an average weatherization measure ex onte savings
estimate of 2,757 kwh, the weatherization measures realized 73% of the expected savings across the
three year period. PY 2011 represents the predominant sample ofthe billing analysis; however, the ex
ante estimates are considerably higher than in other years.20
lf the billing analysis is limited to only PY 2012 and PY 2013 participants, the sample size drops
considerably. Only fourteen 2012 - 2013 participant homes in ldaho passed screening for analysis. The
fixed effects model was unable to estimate savings for this sample. Cadmus therefore presents the
results of our PRISM analysis. Due to the high relative precision of this estimate, Cadmus used the
combined Washington and ldaho sample results for PY 2012 and PY 2013 as the evaluated result for PY
2013 in ldaho. This result is the best estimate of current program performance in ldaho. The combined
PY 2012 and PY 2013 weatherization participants achieved savings of 2,569 kWh, or 11.3% savings
relative to the pre-installation period NAC. With an average weatherization measure ex onte savings
estimate of 1,927 kwh, the weatherization measures realized 133% of the expected savings.
Table tl4 shows the realization rates for the three combined sample analysis groups. The realization rate
of 133% shown for PY 2012 - PY 2013 is used to calculate adjusted gross savings for this program.
EThe previous analysis relied primarily on PY 2O1O participants and resulted in a weatherization savlngs estimate
of 953 kwh with a combined Washington and ldaho realization rate of 35%. PY 2011 savings and reallzation rate
are higher than the PY 2010 estimates. The ex-ante values for PY 2011 participants were developed before our
previous analysis was completed.
52
S.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-I4
Khawaja, The Cadmus Group, lnc
Schedule 1, Page 58 of 130
@
rabre 44. rdaho and *",n,,r,:1,[:::ir::,.,:l;i:f.,..'c savings Rearization Rates
2,375 l7%wA & tD PY 2011-PY 2013
wA & tD PY2012-PY2013* 2,569 3M
u a to pv zou 2,24L 20%
' volues shown in this row ore used os the evoludtion resultsfor PY 2073 in ldoho.
Figure 14 shows a comparison of the weatherization percentage savings to similar electric
weatherization evaluations. Avista's PY 2011, PY 2012 and PY 2013 percent savings have improved
significantly since the PY 2010 program year.
Figure 14. Electric Weatherization Percent Savings Benchmarking
Avista PY 2010 (Previous)
Avista PY 2011WA
Avista PY 2011 lD
Avista PY 2012-2013 WA
Avista PY 2012-2013 lD
Pacificorp (WA 2011)
Pacificorp (lD 2011)
Pacificorp (WA 2013)
Pacificorp (lD 2013)
Puget Sound Energy
12%L4%
Table 45 shows the total reported and qualified counts, savings, and realization rates of electric
weatherization program measures.
53
89%
L31%
80%
6%4%
l7s%
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 1, Page 59 of 130
Table 45. Weatherization Program Results
E Attic lnsulation
with Electric Heat
t iiooiln",tattln
with Electric Heat
E Wall lnsulation
with Electric Heat
I piogr", iotil
24,897
13,t21 100%TW 17,495
100%
tog%
10
37
36,061 o:*
67,U9 fiA7L
118,084 133%
w,471 iii%
100%
1o096
1.3.8. Water Heater Efficiency
Prcgrom Description
The Water Heater Efficiency Program represented one measure: electric high-efficiency water heaters.
Through this program, Avista offered a S50 incentive to residential electric customers who installed an
eligible high-efficiency water heater. Electric water heaters with a tank had to have a 0.93 EF or greater
to qualiry for the program.
Anolysis
The PY 2O1O-PY 2011 electric impact evaluation report2l documented Cadmus' analysis for determining
the change in energy consumption resulting from installing electric high-efficiency water heaters. As
that analysis continues to provide the best information on this measure, we used those results for PY
2013.
Resuls ond Findings
Table 46 shows the total tracked and qualified counts, savings, and realization rate,
21 Cadmus. Ayista 2070-2071 Multi-Sector Electric lmpact Evoluotion Report, May 2OL2.
54
Table 45. Water Heater Efficiency Measure and Reported and Adjusted Savings
S.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
Khawaja, The Cadmus Group, lnc
Schedule 1, Page 60 of 130
@I
1.3.9. ENERGY STAR Homes
Program Description
Avista offered incentives through the ENERYG STAR Homes Program for builders constructing single-
family or multifamily homes complying with ENERGY STAR criteria and certified as ENERGY STAR Homes.
Avista provided a 5900 incentive for homes using electricity from Avista for space and water heating.
Anolysis
ln the PY 2010-PY 2011 electric impact evaluation report, Cadmus documented the simulation modeling
we performed to determine energy savings achieved by ENERGY STAR Homes. As those simulation
results continue to provide accurate estimates of savings, we used those results for PY 2013.
Results ond Findings
Table 47 shows the total tracked and adjusted counts, savings, and realization rates for measures
offered through the ENERGY STAR Homes Program. Avista funded electric measures for participating
Avista homes.
Home-
Electric Only L7,52t 12,550
1.3.10. Geographic CFL Giveaway
Avista gives CFLS to customers at events throughout the year. Avista tracks the number of bulbs
distributed outside oftheir database and separate from the other programs with CFL offerings. Avista
estimates the energy savings as 15 kWh per bulb. This value is conservative compared to estimates
currently in use by the RTF. Cadmus accepts the energy savings estimated using 15 kWh per bulb, and
completed no further evaluation activities.
Low lncome and Senior Citizen Giveaways L,52A
!'r7-9--)
7.4. ResidentiolConclusions
For PY 2013, Avista's residential electric programs produced 5,933,197 kWh in gross savings, yielding an
overall realization rate of 116%. Table 49 shows reported and evaluated gross savings and realization
rates per program.
55
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
Khawaja, The Cadmus Group, lnc
Schedule 1, Page 61 of 130
Table 47. ENERGY STAR Home Program Results
Table 48. Geographic CFL Giveaway Events, Evaluated Savings
S.
Iable 49. Total Program Reported and Evaluated Gross Savings and Realization Rates
Simple Steps, Smart Savings
Second Refrigerator and Freezer
Recycling
ENERGY STAR Products
Heating and Cooling Efficiency
Space and Water Conversions
weatherlzation/lhell
Water Heater fffiiiency
ENERGY STAR Homes
Geographic CFL Giveaway
Program Total
. 3,892,227
2t9,576
79,374
L54,750
658,664
57,Ug
4,496
i,52r
26,640
5,130,507
4,7sO,306
368,r74
29,OLL
t44,iso
506,078
s!,o1
5,487
12,550
26,640
5,933,tg7
722%
168%
37%
gqN
iaN
L33%
lzix
72%
LOO%
tt6%
7.5. ResidentialRecommendotions
Cadmus recommends the following changes to Avista's residential electric programs:
. Avista should consider updating its per-unit assumptions of recycled equipment to reflect this
evaluation in order to ensure that planning estimates of program savings align with evaluated
savings.
o lf Avista chooses to reinstate clothes washer rebates, it should continue to track them all within
the electric program unless there is a large penetration ofgas dryers.
. Avista should increase the measure-level details captured on applications and included in the
database. Specific additional information should include energy factors and/or model numbers
for appliances, baseline information for insulation, and home square footage, particularly for the
ENERGY STAR Homes Program.
. Avista should consider offering tiered incentives by SEER rating, as higher SEER systems
generally require ECM fan motors to achieve the high SEER rating.
Future Reseorch Areas
The following are recommended future research areas for this program. Cadmus based these research
recommendations on the results ofthis impact evaluation and on known future changes to program
requirements.
Avista should consider completing a lighting logger study within its territory if the results of the
forthcoming RBSA study do not accurately represent usage in their territory.
Avista should consider researching the percentage of Simple Steps, Smart Savings bulb
purchases that are installed in commercial settings. This could reflect an increase in the average
installed HoU and increase program savings.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-'|4
Khawaja, The Cadmus Group, lnc
Schedule 1, Page 62 of 130
. Avista should perform a billing analysis of ENERGY STAR Homes using a nonparticipant
comparison group once enough homes have participated under the new requirements to justify
performing the work. This research could be used to demonstrate the savings achieved through
energy-eff iciency construction practices.
. Avista should consider researching the current variable speed motor market activity to
determine ifthis measure should continue as a stand-alone rebate or be packaged with othel
equipment purchases.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 1, Page 63 of 130
2. Residential Behavior Program
2.7. ProgramDescription
For its Residential Behavioral Program, Avista sends home energy reports to residential customers to
educate them about their electricity use and suggest opportunities for saving electricity. Each report
contains:
. An analysis ofthe home's current and past electricity use;
o A comparison of the home's electricity use to the electricity use of its similar neighbors (known
as the neighbor comparison); and
. Electricity savings tips, including promotions of other Avista energy-efficiency programs.
Avista seeks to achieve program electricity savings by increasing awareness of energy efficiency and by
encouraging lasting changes in energy-use behaviors and in the adoption of energy-efficiency measures.
Opower implements the program. Avista expected the program to save about 1% of energy use in PY
2013.
The program was targeted to single-family homes and units in multifamily buildings with above-average
electricity use.22 Although the program is focused on saving electricity, homes that receive electricity
and natural gas service from Avista are eligible to participate. Each home receives six reports during the
first 12 months of the program.
2.1.1. Program Details
The program began in June 2013, when Opower sent the first energy reports to homes in Avista's ldaho
service territory by U.S. mail. Approximately 24,500 Avista ldaho residential electric customers received
one or more reports in 2013. Most program homes received their first report in June or July 2013,
although a small number received theirfirst report in a later month.
To be eligible, homes had to meet the following criteria:
o Have above-average electricity use;
r Have an adequate electricity billing history (12 or more months of continuous bills at the same
premise);
Have a sufficient number of similar neighboring homes (for the neighbor comparison);
Have home occupants who are responsible for payinS electricity bills;
Be a primary residence;
The average annual electriclty use per program home was 16,712 kwh in PY 2012- The median annual energy
use was 15,122 kwh and the 25th and 75th percentiles were 12,395 kwh and 19,429 kwh, respectively.
a
a
a
58
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 1 , Page 64 of 1 30
@I
. Not be master-metered; and
. Have a valid mailing address.
By contacting Avista, a homeowner could stop delivery of the reports at any time; these customers are
referred to as opt-outs. During PY 2013, there were 297 opt-out customers in ldaho, for a rate ol !.?,1%,
which is a very small share of customers that received reports.
Opower implemented the program as a randomized controltrial (RCT), in which Opower identified
homes in Avista's service territory eligible to receive the reports and Cadmus independently and
randomly assigned each home to the program treatment or control group.23 Homes in the treatment
group received the home energy reports while homes in the control group did not receive reports and
were not informed of the program.2o With random assignment, the treatment and control groups are
expected to be equivalent except for the treatment group receiving energy reports, so it is therefore
possible to attribute any difference in average energy use during the program between the groups to
the receipt ofthe reports. RCT is the gold standard in program evaluation, because it yields unbiased
and robust estimates of the program treatment effects. RCT is recommended in the DOE's forthcoming
UMP for Evaluating Behavior-Based Programs (2014) and by State and Local Energy Efficiency Action
Network guidelines for evaluating residential behavior-based programs (2012).25 This approach was also
employed for evaluations of large-scale, home energy reports programs for Washington investor-owned
utilities.2s
Table 50 shows the number of Avista residential customers in ldaho assigned to the treatment group
and the number receiving one or more energy reports in PY 2013. Not every treatment customer
received energy reports because after Cadmus created the random assignments, Opower determined
that some customers did not have a valid mailing address or were missing information required to
generate a report. The table also shows the total number of customers in the control group and the
Using standard statistical tests, Cadmus verified that the treatment and control groups were balanced in terms
oftheir annual, summer, and winter ADCS.
Opower could not deliver reports to a small number of homes assigned to the treatment group, as discussed
later in this report. Opower also identified control homes for which it would have been impossible to send a
home energy report.
See: State and Local Energy Efficiency Action Network. Evdluotion, Medsurement, ond Verificotion (EM&V) of
Residential Behdvior-Bosed Eneryy Efliciency Programs: lssues and Recommenddtiors. Prepared by A. Todd, E.
Stuart, S. Schiller, and C. Goldman, Lawrence Berkeley National Laboratory. 2012. Available online:
Also see the draft DOE UMP protocols for evaluating behavior-based
programs:
Seei Puget Sound Energy's Home Energy Reports Progrom. Prepared by DNV KEMA Energy & Sustainability
2012. Available online: https://conduitnw.orEl lavouts/Conduit/FileHandler.ashx?RlD=849
59
S.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
Khawaja, The Cadmus Group, lnc
Schedule 1, Page 65 of 130
number of customers in the control group who would have received reports ifthey had instead been
assigned to the treatment 8roup.
Randomlyassigned 25,2@ 13,000
Randomly assigned and receiveJ a report (treatment) or
could have recelved a report (control) ' 24'50L 12'630
38,200
37,L3t
* This row excludes treatment homes that did not receive a report and control homes that could not have
received a report due to an invalid mailing addres or unavailable information required to generate a report.
2,2. Residential Behavior Program lmpoct Evaluation Methodology
For the impact evaluation, Cadmus estimated the program energy savings in PY 2013 and quantified the
program impact on participation in Avista's other residential efficiency programs, Cadmus used a panel
regression analysis of customer monthly bills to estimate the program's electricity savings between
meiling of the first reports in June 2013 and December 2013. Cadmus analyzed Avista efficiency program
participation and measure savings data to estimate the program's effects on participation in other
Avista efficiency programs, as well as to estimate savings that were counted towards other efficiency
programs,
More specifically, Cadmus' evaluation of the Residential Behavior Program savings and efficiency
program uplift consisted of the following fourtask:
1. Data collection, review, and preparation.
2, Equivalency analysis (checks on treatment and control groups).
3. Billing analysis.
4. Energy-efficiency program uplift and savings analysis.
2.2.1. Data Collection, Review, and Preparation
To perform the billing and uplift analyses, Cadmus collected the data outlined below.
Monthly Customer Bills
Avista supplied Cadmus with monthly electricity and gas bills (for dual-fuel customers) between June
2012 and January 2014. The data included approximately 12 months of bills prior to and six months of
bills after the program began for homes in the treatment and control groups. These billing data
included: account numbers, energy use during the monthly billing cycle, number of days in the billing
cycle, and the first and last days of the billing cycle.
60
Table 50. Number of Treatment and Control Homes in PY 2013
S.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-I4
Khawaja, The Cadmus Group, lnc
Schedule 1, Page 66 of 130
Progrom lnformation
Cadmus obtained program enrollment information from Opower. These data included the following
fields for each home in the treatment and control Sroups:
a
a
a
a
a
Address of residence;
Assignment to treatment or control group;
Date first report was generated ;27
Opt-out date for homes in the treatment group choosing not to participate in the program;
lnactive date for homes that closed their gas or electric account; and
Account numbers (for linking to billing data).
Weother
Cadmus collected daily average temperature data for weather stations in the program region from the
National Climate Data Center (NCDC). For a small number of stations where the NCDC data were
incomplete, Cadmus was able to interpolate the daily average temperature as an average of the
preceding and following day. ln cases where a string of days were missing data, Cadmus used
temperature data from the next-nearest weather station. Then we used temperatures to calculate the
number of HDDs and CDDs for each customer billing cycle.
Residentiol Energy-Efficiency Progrom Trocking Doto
Avista provided Cadmus with participant and measure savings data for any PY 2013 residential energy-
efficiency programs in which participation could have been influenced by the behavior program. These
programs included those offering appliance recycling and residential rebates for HVAC equipment,
conversions to natural gas, and insulation.
For each program and measure, the data included: the account number; the numberand description of
measures installed; measure installation dates; and verified gross savings. Cadmus used this information
to estimate the Residential Behavior Program's participation and savings effects on other efficiency
programs.
Doto Cleoning
Cadmus conducted a number of steps to inspect and clean the data provided by Opower. The steps are
described in Appendix B: Residential Behavior Program Data Cleaning Procedures. Cadmus did not
identify any significant issues with the Opower data.
Cadmus requested monthly billing data from Avista for ldaho customers from June 2012 through
February 2014. Avista provided bills for all but a few customers in the program treatment and control
2' Opower assigned a pseudo first report date to control homes, representing the date the first energy report
would have been mailed.
67
Exhibit 3
Gase Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 1, Page 67 of 130
groups.a Cadmus then followed a number of steps to clean the billing data. These steps are also
described in Appendix B: Residential Behavior Program Data Cleaning Procedures.
Dota Preporution
Using the number of days in the billing cycle, Cadmus expressed each month's energy use and weather
in average daily terms, then merged the billing, weather, and program information data, includinS
information about the approximate delivery date of the first home energy report.
Cadmus performed billing analysis on the population of program homes, except for homes from the
estimation sample that satisfied one or more of the following criteria;
o The home was in the treatment group but did not receive a home energy report or was in the
control group but would not have received a home energy report (indicated by the customer
information data missing the first report date).2s
o opower flagged the home as receiving a home energy report, but the home had not been
randomly assigned to the treatment group,30
r The home did not have a complete or near-complete billing history forthe 12 months before the
start of the program. Cadmus dropped homes from the analysis that had fewer than 11 bills
between June 2012 and May 2013.
Applying these filters resulted in a group containing 3d382 customers: 11,730 in the control group and
22,652 in the treatment group. Although the billing analysis excluded homes with fewer than 11 bills in
the year before the program, the savings estimate includes savings from these homes.31
2.2.2. Equivalency Analysis
Per an agreement between Avista, Cadmus, and Opower, Cadmus randomly assigned eligible residential
customers to the program treatment and control Sroups. At that time, Cadmus verified that the random
assignment resulted in treatment and control groups that were balanced in terms of their annual,
winter, and summer electricity use. Cadmus provided these random assignments to Opower, who
Avista provided billinS data for all but 858 customers (315 in ldaho). While we did not use these customers'
bills in the savings analysis, we did count the savings from these customeE in our estimated PY 2013 total
program savings.
A home in the treatment group may have been missing a first report date because either the account became
inactive before the first report was generated, or Opower did not have a valid mailing address. An
approximately equal number of control homes were not assigned a first report date and were left out ofthe
analysis for the same reasons.
For example, this group included utility employees who requested to participate in the program.
Cadmus followed guidelines in the State and Local Energy Efficiencl Action Network report, EM&V of
Residential Behavior-8ased Energy Efficiency Programs (2012), to drop homes with less than 10 months of
billinS data from the analysis.
s
31
62
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 1, Page 68 of 130
additionally analyzed them using proprietary home and demographic characteristic data and verified
that the groups were balanced.
Cadmus also performed an equivalency check of homes in the treatment and control groups after
applying the filters described in the preceding section. As Table 51 shows, the difference between the
two groups' annual consumption is small and not statistically significant,
Treatment
Control
t value
t6,7ro
L6,7t4
o.05
P value 0.95
As described below, any time-invariant differences in energy use between the treatment and control
groups after filtering are absorbed with customer fixed effects.32
2.2.3. BillingAnalysis
To estimate Residential Behavioral Program electricity savings, Cadmus used difference-in-differences
(D-in-D) regression. D-in-D regression usesthe energy use oftreatment and control group homes before
and after the first energy reports to account for any naturally occurring efficiency that might have been
correlated with Residential Behavior Program activity.
The D-in-D approach requires monthly energy use from before and during the program in the treatment
and control group homes. Using Avista billing data, Cadmus conducted panel regression analysis ofthe
electricity consumption in ldaho to estimate the average program savings per home per day in PY 2013.
Model Specification
The average daily consumption (ADC) of electricity in home 'i' in month t' is given by:
ADCit = 91 POST1 + p2 PART; x POSTft + W'y + o, + t, + r,,
A home fixed effect represents the portion of a home's energy use that does not vary over time. This energy
use is captured in the regression analysis by the inclusion of a separate intercept for each customer or by
equivalently transforming all the variables by subtracting home-specific means.
53
S.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
Khawaja, The Cadmus Group, Inc
Schedule 1, Page 69 of 130
Where:
POST
Coefficient representing the impact of non-program factors on
consumption between pre-program and program months.33
An indicator variable for whether the month is pre- or post-treatment.
This variable equals 1 in months following the first report date and 0
otherwise. The variable is defined with a short lag to allow for time
between the report's generation and delivery to the home.s
Coefficient representing the conditional average treatment effect (ATE)
of the program on electricity use (kWh per home per day).
An indicator variable for program participation (which equals 1 if the
home was in the treatment group, and 0 otherwise).
A vector using both HDD and CDD variables to control for the impacts of
weather on energy use.
Vector of coefficients representing the average impact of weather
variables on energy use.
Average energy use in home 'i'that is not sensitive to weather or time.
Analysis controlled for non-weather-sensitive and time-invariant energy
use with home fixed effects.
Average energy use in month t' reflecting unobservable factors specific
to the month. The analysis controls for these effects with month-by-
year fixed effects.3s
Error term for home 'i' in month 't.'
PART
Progrom Energy Sovings
Cadmus estimated the total Residential Behavioral Program energy savings in PY 2013 by multiplying the
total number of program days across treated homes by the average savings per home per day, pr. To
illustrate, leti=L,2,..., N index the number of homes receivin8 a home energy report; and D(x) return
the number of the days in 2013 from January 1 for a given date x (e.g., D(February 1)=32).
ln addition to naturally occurring efficiency, this coefficient captures differences in average consumption
between pre-program and program months due to having 12 months of pre-program bills and only seven
months of program bills.
Specifielly, we defined the first report date as 14 days after the report was Senerated to allow time for report
delivery.
Cadmus included month-by-year fixed effects and POST in the same model because there was variation
between customeB in the month ofthe first report date.
F,
9,
W
t1
tfr
il
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 1, Page 70 of 130
E@I
The net program savings then equa',iL
r.r,"r, = -u*rer* prosDavsrl
Where:
i = 1,2,...,N; indexes the number of homes in the treatment group.
ProgDaysl = 365 - D(first report datei), if the billing account for home 'i' was still
active on December 31, 2013, and,
D(inactive dater) - D{first report date), if the billing account for home 'i'
became inactive before December 31, 2013.
As the definition of ProgDaysi shows, Cadmus counted savings from treated homes whose accounts
became inactive up until the accounts closed.
2.2.4. Energy-Efficiency Program Uplift and Savings Analysis
The Residential Eehavioral Program could have increased participation in Avista's other efficiency
programs in two ways:
o First, energy reports directly educated customers about some ofAvista's efficiency programs
and encouraged them to take advantage of program offerings and incentives.
. Second, the reports could have raised customer awareness and knowledge of energy efficiency,
which may cause some to participate in Avista's efficiency programs.
Analysis of efficiency program uplift is important for two reasons:
. First, Avista sought to learn whether and to what extent the Residential Behavior Pro6iram
caused participation in its other efficiency programs.
o Second, to the extent the Residential Behavioral Program caused participation in other
efficiency programs, energy savings resulting from this participation will have be counted twice:
in the regression estimate of Residential Behavior Program savings, and in the other programs'
savings. (Thus, Avista will want to subtract the double-counted savings from its portfolio
savings.)
The uplift analysis described here yields estimates of the effect of the Residential Behavioral Program on
other efficiency program participation and the amount of double-counted savings. The analysis was
limited, however, to program measures that Avista tracked at the customer level, and thus did not
include residential upstream programs promoting cFLs through store discounts, However, analysis of
65
S.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
Khawaja, The Cadmus Group, lnc
Schedule 1, Page 71 of 130
&fl
Opower home energy report programs in other service territories suggests that CFLS account for only a
small percentage oftotal program savings.36
Methodology
As with the energy-savings analysis, for the uplift analysis Cadmus followed the logic of the program's
experimental design. Cadmus collected Avista electric efficiency program participation and savings data
for PY 2013, matched the data to the program treatment and control homes, and estimated uplift as a
simple difference in participation rates and savings between treatment and control groups. As
customers in the treatment and control groups are expected to be similar, except for having participated
in the behavior program, the difference between treatment and control groups in other efficiency
program participation is expected to equal the true Residential Behavior Program uplift. ln matching
treatment and control homes to the PY 2013 efficiency program data, Cadmus excluded measures
installed after an account became inactive or before the first energy report date.
Let p. be the participation rate (defined as the number of efficiency program participants to the number
of potential participants) in a PY 2013 program for group m (as before, m=1 for treated homes, and m=0
for control homes). Then:
Participation uPlift = pr-po
Expressing participation uplift relative to the participation rate of control homes in PY 2013 yields an
estimate of the percentage of uplift:
% of participation uplift = program uplift/p6
Residential Behavior Program savings from participation in other efficiency programs can be estimated
the same way, by replacing the program participation rate with the program net savings per home:
Net savings per home from participation uplift = o1-oe.37
Multiplying net savings per home from participation uplift by the number of program homes yielded an
estimate of the total Residential Behavioral Program net savings counted in Avista's other efficiency
programs.
s€e the impact evaluation of Pacific Gas & Electric's Home Energy Reports Program, 2010-2012, which is
available online:
2013 CALMAC lD PGE0329.01.pdf
cadmus obtained net savings by multiplying measure-verified gross savings by the estimated measure net-to-
tross (tlTG) ratio.
66
S.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
Khawaja, The Cadmus Group, lnc
Schedule 1, Page 72 ot '130
Cadmus performed participation and savings uplift analyses for the following Avista residential efficiency
programs:
o Second Refrigerator and Freezer Recycling
. Residential rebate programs, including:
. Space and Water Conversions (conversion from electric furnace to NGF or electric water
heater to NGWH)
. Heating and cooling Efficiency (ASHPs (includlng conversions), variable speed motors, and
electric water heaters)
. weatherization/Shell (floor and attic insulation)
Cadmus did not perform uplift analyses for the following residential electricity efficiency programs:
. Geogruphic CFL Gfueoway.fhough the Residential Behavior Program may have influenced CFL
and other high-efficiency lighting purchases, such purchases were tracked at the store level.
o ENERGY STAR Homes. This program targeted builders of new homes, which the Residential
Behavior Program did not target.
2.3. Program Results qnd Findings
2.3.1. Electricity Savings per Home Estimates
Table 52 shows the average daily energy savings per home or, equivalently, the conditional ATE per
home of Avista's Residential Behavioral Program. The savings are represented by the coefficient on the
interaction variable PARTI x POSTit. On average, homes saved 0.674 kWh (1.57%) per day.* This savings
estimate was statistically significant at the 1% level.
For perspective, these savings could be achieved by turning off a 65-watt incandescent lamp for 10
hours per day or by replacing nine 100-watt incandescent lamps used for one hour each day with nine
25-watt cFl-s.
Average svings of 1.57% during the first seven months is slightly Breater than the average savings over the
same period estimated for other utility home energy reports pro8rams. See: Allcott, Hunt. (2011). Social
Norms and Energy Conservation. Joumal of Public Economi6, 95(2), 1,082-1,095. Also see: Rosenberg;
Mitchell, G. K. Agnew, and K. Gaffney. Causality, Sustainability, and Scalability - What We Still Do and Do Not
Know about the lmpacts of Comparative Feedback Programs. Paper prepared for 2013 lnternational Energy
Program Evaluation Conference, Chicago, lllinois, August 13-15, 2013.
67
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 1, Page 73 of 1 30
Table 52. conditional Average Treatment Effect*
0.674PARTt x PosTft - Year 1 (Year 1 savings per day Per home)(o.oes)
Customer fixed effects Yes
Y:q+rretlrg{:$q, --L , I-"t --:- - --_:- -,J,ilt- -lN (h91"r) _. The dependent variable is average daily electricity use in the month for a treatment or control Broup home. The
model estimated this by ordinary least squares using monthly bills between June 2012 and January 2014. Huber-
White estimated standard errors (shown in parentheses) are clustered on homes.
Cadmus ran several other model specifications to verify the robustness of the savings estimates with the
inclusion or omission of different variables. For example, we estimated models with and without
different combinations of home-fixed effects, time-fixed effects, and the weather variables. Appendix C:
Residential Behavior Program Regression Model Estimates includes complete results from these other
regression specifications. Little or no difference occurred in the estimated savings between
specifications-an expected result, as estimates of treatment effects in large RCTS typically prove robust
to changes in model specifications.
Table 53 shows the average savings per Residential Behavior Program home in PY 2013. Cadmus
obtained this estimate by multiplying the estimated savings per home per day in Table 52 by the average
number of program days for treated homes in PY 2013. We defined the program days for a home as the
number of days between the first report date and December 31, 2013.
. Cadmus estimated these savings per home based on Table 52 and on the average number of program days per
home in PY 2013.
Figure 15 shows estimates of average savings per month from June 2012 to January 2014. Cadmus
obtained savings via a regression that estimated the difference in energy use between treatment and
control group homes, conditional on home fixed effects. The ATE is shown as a percentage of the ADC of
control group homes.
lt.ff)s} l
Yes
68
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 1, Page 74 ot 130
Table 53. Average Savings (kWh) Per Home for PY 2013*
Figure 15. Average Savings Per Month*
oJere""d,rf odgf od.rf of*f J'.u9 J+f /.ro**"J
t Cadmus obtained the savings estimates in this figure from a regression of ADC on home fixed effects, month-by-
year fixed effects, and month-by-year fixed effects interacted with an indicator of whether that home was in the
treatment group. As the model also includes home fixed effects, it was necessary to omit one month-by-year fixed
effect.
As expected, there were not significant differences in average energy use between treatment and
control group homes before Opowersentthe first energy reports in June 2013. The 90% confidence
interval includes zero in each month. The approximate equality of energy use before treatment means
that we cannot reject the identifying assumption ofthe savings analysis: that receiving a home energy
report was random and uncorrelated with expected energy use.
Treated homes started saving energy after receiving the first reports. ln July and August, percentage
savings were below 1% but still substantial. Percent savings increased in subsequent months. The
ramping of savings in the first six months of the program is evident in Figure 15, which is typical of home
energy report programs.
2.3.2. Program Savings Estimates
Table 54 reports the total program savings for Avista's ldaho service territory. Cadmus estimated savings
by multiplying the estimate of average daily savings per home by the total number of program days for
treated homes.
69
S.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-I4
Khawaja, The Cadmus Group, lnc
Schedule 1 , Page 75 of 1 30
Table 54. Residential Behavioral Program Energy Savings in PY 2013
ldaho l.2Wo 1.57%2,925,8@ 2,224,203 3,507,516 r3t%
r Cadmus obtained ex onte percentage electricity savings from the 2013 Avista Business Plan. Avista expected
1.4% electric savings from the program in the first year, and assumed that 40% of the first-year eneryy savings
would occur in the first six months ofthe program in 2013. Given the 2013 consumption data for the control
group, it follows that the savings expected for the first six months ofthe progrem are L.2Yo. Evaluated annual net
elstricity savinSs are based on the savings estimate shown in Table 53.
Avista expected net savings of 1.2% from the Residential Behavioral Program in PY 2013. Based on the
regression analysis of monthly energy use, Cadmus determined that the program achieved net savings
ol L.57%. Cadmus estimated net savings of 2,925,860 kwh in PY 2013, with a 90% confidence interval
12,224,203 kwh, 3,507,516 kwhl, or relative precision of t23%. The program realized 131% of the
expected savings.
2.3.3. Uplift Analysis
This section reports estimates ofthe Residential Behavioral Program's effect on participation in Avista's
other efficiency programs (the uplift), as well as savings resulting from additional participation. To avoid
double-counting savings, behavior program savings from participation in other efficiency programs must
be subtracted from the residential portfolio savings. ln estimating participation uplift and savings from
uplift, Cadmus considered onlythose measures installed afterthe first reports were received.
Table 55 shows the percentage uplift estimates for each program. As noted in the methodology, uplift
equals the absolute effect on the participation rate, and the percentage uplift equals the participation
rate effect divided by the participation rate of control homes in PY 2013.
Second Refrigerator and Freezer
Recyclint 41%
Resldentlal Rebate Protrams
space and wateiconversions 0.01%
Heating and Cooling Efficiency O.2l% 100%
Weatherization/shell OOl% 158%
* Participation uplift is an estimate oi.trange in ttte ,ate ot piogra. patticipation attriUuiaUteio ttre nesidential
Behavior Program. The percentage of participatlon uplift is the change in the participation rate relative to the
program participation rate of customers in control homes in PY 2013. The text below provides estimation details
and data sources.
lix
70
Table 55. Residential Behavioral Program Participation Uplift+
S.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
Khawaja, The Cadmus Group, lnc
Schedule 1, Page 76 of 130
The Residential Behavioral Program increased the rate of participation of customers in the Second
Refrigerator and Freezer Recycling, Space and Water Conversions, Heating and Cooling Efficiency, and
Weatherization/Shell programs. While this increase was less than 1%, the baseline rate of participation
was relatively low, so the percentage uplift effect was higher, especially for the Weatherization/Shell
Program.
The Second Refrigerator and Freezer Recycling Program experienced 41% uplift, the Space and Water
Conversions Program experienced 16% uplift, the Heating and Cooling Efficiency Program experienced
100% uplift, and the Weatherization/Shell Program experienced 158% uplift.3e This means, for example,
that treatment homes were 41% more likely to partlcipate in the Second Refrigerator and Freezer
Recycling Program than control homes.
Savings Anolysis
Table 55 shows electricity savings from uplift in participation in the Second Refrigerator and Freezer
Recycling Program and the residential rebate programs in PY 2013. The savings reflect the behavior
program's effects on both participation rates and on the numbers and/or kinds of measures installed.@
The savings from program uplift reported in Table 55 should be subtracted from the PY 2013 residential
portfolio savings.
Second Refrigerator and Freezer
Recycling
Residential Rebate Programs
lpry r"!l{. !:, co1y"r:io.n:...-
Heating and Cooling Efficiency
3,238
30,700
0.13- 12i
Weatherization/shell
iota-
0.08 L,957
s49s52.74
Participation in the Residential Behavior Program resulted in Avista efficiency program savings of 54,955
kWh, equal to 1.9% of the behavior program savings. The majority of uplift savings derived from
The percentage uplift for the Weatherization/Shell ProBram was large because the increase in the conversion
rate was large relative to the baseline rate.
The methodology called for using net savings of efficiency measures in calculating Residential Behavioral
Program savings from efficiency program uplift; however, except for the Second Refrigerator and Freezer
Recycling Program, Cadmus did not derive NTG values for program measures. lnstead, we used adjusted gross
savings estimates based on field estimates of utilization and installation rates to calculate uplift savings. For
consistencl across programt we used the adjusted gross savings for the Second Refrigerator and Freezer
Recycling Program.
71
Table 55. Residential Behavior Program Electricity Savings from Program Uplift
S.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-'|4
Khawaja, The Cadmus Group, lnc
Schedule 1, Page 77 ot 130
residential conversions of electricity to gas. To avoid double counting the savings from uplift must be
subtracted from evaluated savings for the electricity efficiency portfolio, from the Residential Behavior
Program, or from other efficiency PY 2013 programs.
2.3.4. Evaluated Net Savings Adjustment
Table 5TErrorl Reference source not found, shows the Residential Behavioral Program adjusted net
savings for PY 2013. The adjusted savings are the difference between the program-evaluated net savings
and estimated savings from program uplift. The adjusted net proSram savings in PY 2013 were 2,870,905
kwh.
2.4. Residential Behovior Progrom Conclusions
Analysis of the monthly electric bills of treatment and control homes during the first seven months of
the Residential Behavior Program led to the following PY 2013 findings:
. Homes in ldaho saved an average 0.674 kwh (1.57%) per day. The percentage savings were
hi8her than expected (1.2%).
o The program achieved total electricity savings of 2,925,860 kwh. The relative precision of the
electricity savings estimate was j23% with 9096 confidence.
. The program generated percentage savings at a slightly higher rate than the normal range for
ener8y reports programs.
Analysis of Avista's energy-efficiency program data resulted in the following findings about the
Residential Behavior Prognm effects on other efficiency program participation and savings:
. The Residential Behavior Program lifted the rate of participation in the Second Refrigerator and
Freezer Recycling, Space and Water Conversions, and Weatherization/Shell programs. The
percentage uplift for the Space and Water Conversions Program was large because of the low
baseline rate of conversions.
The total Residential Behevior Program electricity savings from efficiency program uplift was
54955 kWh, or 1.9%.
Savings from efficiency program uplift are counted in the Residential Behavior Program
regression-based estimate of savings and in other programs' savings. To avoid double counting
the uplift savings must be subtracted from the evaluated savings for the electric portfolio or for
the Residential Behavior Program.
After adjusting net electricity savings for program uplift, the program saved 2,870,095 kwh.
72
Exhibit 3
Case Nos. AVU-E-l4 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 1, Page 78 of '130
Table 57. Residential Behavioral Program Adjusted Net Savings in PY 2013
2.5. Residential Behovior Progrom Recommendotions
Based on the analysis, Cadmus makes the following recommendations:
Avista should continue to promote its efficiency programs in the energy reports, as the reports
increased both the rate of efficiency program participation and savings.
Avista should consider performing additional research about the peak-coincident demand
savings from the Residential Behavior Program to determine whether it is cost-effective relative
to existing residential load control programs.al
Research would require analysis of high frequenry (15 minute or one hour interual) energy use data for a large
number of treatment and control group homes. For an example of such an analysis, see: Stewart, James. Peok
Coincident Demand Savingslrom Residentiol Behovior-Bdsed Progrums: Evidence lrum PPL Electric's Behavior
ond Educdtion Progrom. 2013. Available at htto://escholarshio.orsluc/item/3cc9b3ot.
73
S.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
Khawaja, The Cadmus Group, lnc
Schedule 1, Page 79 of 130
3. Nonresidential lmpact Evaluation
3.7. lntroduction
Through its nonresidential portfolio of programs, Avista promotes the purchase of high-efficiency
equipment for commercial utility customers. Avista provides rebates to partially offset the difference in
cost between high-efficiency equipment and standard equipment.
The nonresidential electric portfolio has 11 programs in three major categories: prescriptive programs,
the Energy Smart Grocer Program, and the Site-Specific Program (for custom projects). These programs
are described below.
Prescriptive Commerciol Clothes Washer
To encourage customers to select high-efficiency clothes washers, this program is targeted to
nonresidential electric and natural gas customers in multifamily or commercial Laundromat facilities.
Avista streamlined the program approach to reach customers quickly and effectively and to promote
ENERGY STAR or Consortium for Energy Efficienry (CEE)-listed units.
Prescriptive Commerciol Windows and lnsulation
Beginning in January 2011, Avista has processed the installation of commercial insulation through this
prescriptive program in addition to the Site-Specific ProSram. Projects are eligible for the Prescriptive
Commercial Windows and lnsulation Program when they have:
o Wall insulation of less than R4 that is improved to R-11 or better
. Attic insulation of less than R-11 that is improved to R-30 or better
o Roof insulation of less than R-11 that is improved to R-30 or better
P rescri ptive F oo d Se rvice
Applicable to nonresidential electric and Eas customers with commercial kitchens, Avista provides direct
incentives to customers who choose high-efficiency kitchen equipment though this program. The
equipment must meet either ENERGY STAR or CEE tier levels (depending on the unit) to qualify for an
incentive.
Prescriptive Green Motors lnitiative
Operated in partnership with The Green Motors Practices Group*, Avista provides education through
this program to foster the organization and promotion of member motor service centers' commitment
to energy-saving shop rewind practices for motors ranging from 15 HP to 500 HP.
a htto://www.greenmotors.orsl
74
Exhibit 3
Case Nos. AVU-E-l4 AVU-G-14
Khawaja, The Cadmus Group, lnc
Schedule 1, Page 80 of 130
S.
@
Prescriptive Lighting
Since there is a significant opportunity for lighting improvements in commercialfacilities, Avista offers
direct financial incentives to customers who increase the efficiency of their lighting equipment through
this program, The rebate is available to existing commercial and industrial electric customers whose
facilities are on rate schedules 11 or above. Avista provides pre-determined incentive amounts for 38
measures, including:
o T12 fluorescent to T8 fluorescent lighting
o High bay, high-intensity discharge lighting to T5 fluorescent or T8 fluorescent
. High bay, high-intensity discharge lighting to induction fluorescent
. lncandescent to cFL or cold cathode fluorescent
. lncandescent to LED
. lncandescent exit si8ns to LED exit signs
Prescriptive Motor Contrcls HVAc
The use of single-speed motors to drive fans or pumps often provides the opportunity to save energy
through the use of a variable frequency drive (VFD). A VFD cen convert a single-speed motor to a
variable speed motor with no modification to the motor itself. This can be an efficient way to convert
constant volume air systems into variable volume systems, for example. VFDs are readily available for
motoB from 1 HP to 300 HP and are easily installed directly into the power line leading to the motor,
replacing the existing motor starter. Avista provides incentives for the installation of VFDs.
Many fan and pump systems have a cost-effective application for VFD5. Quite often these systems have
a variable flow rate through the use of throttlin8 devices, such as valves and dampers that vary the flow.
Throttling devices essentially waste excess energy to meintain a given pressure or flow, and the use of a
VFD can be very cost-effective in these situations. Typical examples of systems using throttling devices
are: booster pumps for domestic water, process chilled or condenser water systems, and fan discharge
dampers,
Other variable flow systems use mechanical or electrical methods such as inlet vanes, outlet dampers,
eddy current clutches, hydraulic couplings, or variable pitch pulleys to vary the speed of the fan or
pump. These are more efficient than throttling devices, but not as efficient as VFDs. Some fan and pump
systems that currently have a constant flow may be converted to variable flow through system
modifications.
Prescriptive PC Network Controls
Computers that remain in a full-power state when idle can waste significant energy, especially for
customers with numerous PCs. Through this program, available to nonresidential electric customers,
Avista provides an incentive for the installation of a network-based power management software
solution that manages the power of networked PCs.
75
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 1, Page 81 of 130
Prcscriptive Stondby Generator Block Heater
Most block heating technology employs natural convection within the engine block system to drive
circulation-more commonly known as thermosiphon. Avista promotes the replacement of
thermosiphon-style engine block heaters with pump-driven circulation units, which reduces the overall
block temperature. Because this replacement also decreases the heat transfer rate from the block to the
environment, it can reduce overall block heater energy consumption, which is tied to the circulation
method.
Because thermosiphon heaters require temperature variation to drive circulation, warmer coolant rises
to the top of the block and colder coolant descends to the lower sections of the block. The coolant in the
lower portions of the block must meet the minimum block temperature requirements, which means the
coolant in the upper parts of the block will exceed the minimum temperature requirements. A pump-
driven heater does not require a temperature difference to drive flow, leading to a more uniform
coolant temperature throughout the block. This reduces the overall average block temperature and
minimizes the driving force affecting heat transfer.
Renewables
Avista provides prescriptive incentives for residential and nonresidential projects installing photovoltaic
(solar electric) systems and/or wind turbines,
Energy Smart Grocer
Refrigeration has high potential for energy savings, but is often overlooked because of the technical
aspects of the equipment. Through the Energy Smart Grocer Program, Avista assists grocery store
customers with technical aspects of their refrigeration systems, while also providing guidance as to the
amount of savings they can achieve. A field energy analyst offers technical assistance to customers,
produces a detailed report of the potential energy savings at their facility, and guides them through the
program process from inception through the payment of incentives for qualirying equipment.
Site Specilic
The Site-Specific Program is for nonresidential measures that are not addressed by any of the
prescriptive applications, but must be considered based on their project-specific information. For a
measure to be considered, it must demonstrate kwh and/or therm savings. These measures are
available to all commercial, industrial, or pumping customers that receive electric or natural gas service
from Avista.
Electric and saving measures included in the program are:
. Site-Specific HVAC
. HVAC Combined (heating and cooling)
. HVACCooling
. HVAC Heating
. Multifamily Measures
76
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
Khawaja, The Cadmus Group, lnc
Schedule 1, Page 82 of '130
S.
Site-Specific Lighting
. Lighting Exterior
' Lighting lnterior
Site-Specific Other
' APP|iances
' Compressed Air
. Green Motors Rewind
. lndustrial Process
. Motor Controls lndustrial
. Standby Generator Block Heater
. site-Specific Shell
Avista implements the Site-Specific Program and prescriptive programs, while PECI implements the
Energy Smart Grocer Program. As implementers, both Avista and PECI are responsible for designing and
managing program details. Both implementers developed algorithms for use in calculating measure
savings and determining measure and customer eligibility.
Avista staff fields inquiries from potential participants and contractors and maintains a project tracking
database. Throughout the program, Avista manages projects by reviewing and approving applications at
all stages of the process, calculating project savings, and populating the database with relevant
information.
i.2. Methodology
Cadmus designed the impact evaluation to verify reported program participation and estimate energy
savings. For the impact evaluation, we determined gross savings through engineering calculations,
verification site visits, metering, and some project-level billing analysis.
We reviewed Avista's reported gross energy savings and available documentation, such as audit reports
and savings calculation work papers, for a sample of sites, giving particular attention to the calculation
procedures and documentation for savings estimates. We also verified the appropriateness of Avista's
analyses to calculate savings, as well as the operatang and structural parameters of the analyses. We
then determined gross evaluated energy savings through site visits and engineering calculations for a
sample of projects.
Cadmus collected baseline, tracking, and program implementation data through on-site interviews with
facility staff. During on-site visitt we verified measure installations and determined any changes to the
operating parameters since the measures were first installed. We also interviewed facility staff about
their experiences and any additional benefits or shortcomings of the installed system. We used the
savings realization rates from site visits to estimate savings and develop recommendations for future
studies.
77
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
Khawaja, The Cadmus Group, lnc
Schedule 1 , Page 83 of 1 30
S.
3.2.1, Sampling
Cadmus developed a sampling calculation tool to estimate the number of on-site visits required to
achieve the rigor precision target shown in Table 58. We used preliminary program population data
provided by Avista, and determined we needed to conduct measurement and verification at 107 sites.
We anticipated achieving 90/10 precision for the overall nonresidential portfolio level through the
tartets for each stratum.
Prescri ptive 90120
90120
90l2O
Site-Specific Lighting
IsiGjpi.ir"-ott'"t
site-specific Shell-roui
Cadmus selected both a census and random sample from each stratum. The census projects represented
a small number of participants in the stratum with large savings impacts, The cutoff for the census
savings for each stratum is shown in Table 59. We visited all census project sites. Within each stratum,
we also randomly selected additional site visits from the remaining population of projects.
Prescriptive
15
7
t(t
Site-Specif lc Lighting
Site-speciflc Other 5C1O,000
Site-Specific Slell N/A
Table 60 shows the precision achieved for the actual number of evaluation activities for electric
measures. ln subsequent sections ofthis report, we explain the differences between our initial proposed
and actual sampling plan for the evaluation activities. For example, in our initial sampling plan we
categorized ENERGY STAR appliances in the 'Site-Specific Othe/ category, but as the impact evaluation
progressed, we determined these measures were more appropriate for the 'Prescriptive' category.
300,m0
-,--,eqqq,-__ 5oo,ooo
]500,000
Energy Smart Grocer
site-Specific HVAC
78
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 1, Page 84 of 130
Table 58. Proposed PY 2012-PY 2013 Nonresidential Evaluation Activities
Table 59. Census-Level Cutoff by Stratum
E@
Table 60. Final PY 2012-PY 2013 Electric Evaluation Activity Sample
Prescriptive
Energy Smart Grocer
Site-Specific HVAC
Site-Specific Lighting
Site-Spe€ific Other
Site-Specific Shell
Total
90lL?
sols
9016
golTr
so;13
so/7r
9019
7
2
1
t.,
7
o
22
ln selecting the random sample from each stratum, we found that the extract from Avista's database did
not include addresses that would enable us to identify whether projects performed for the same
company were at different sites, nor did it include information on the specific measures installed.
Therefore, our sampling process was iterative. From the extract, we determined the final primary and
backup samples by selecting projects of interest and asking Avista for additional data, which we received
and used to determine the number and types of projects at various locations.
Also, the database extract provided programievel data, but not measure-level information. Therefore,
we attempted to verify savings for every incented measure at each site, regardless of whether it
achieved gas or electric savings. We were unable to determine whetherwe evaluated an accurate
distribution of measure types withln each program, which would have required an exhaustive review of
project files and it was not within the scope of the evaluation.
3.2.2. DataCollection
Cadmus collected metering data from 22 sites and conducted verifications at 120 sites. For each, we first
conducted a document review to determine measure type, quantity, operational parameters, and
calculation methodology.
Document Review
Avista provided Cadmus with documentation of the energy-efficiency projects undertaken at the sample
sites. We reviewed program forms, the tracking database, audit reports, and savings calculation work
papers for each rebated measure. ln reviewing calculation spreadsheets and energy simulation models
relevant to the evaluation effort, we paid particular attentlon to calculation procedures and
documentation for savings estimates.
79
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
Khawaja, The Cadmus Group, lnc
Schedule 1, Page 85 of 130
S.
Cadmus reviewed each application for the following information:
. Equipment being rcploced: descriptions, schematics, performance data, and other supporting
information.
o New equlpment instolled: descriptions, schematics, performance data, and other supporting
information.
o Savings colculotlon methodology.' methodology used, specifications of assumptions and sources
for these specifications, and correctness of calculations.
Short-Term Metering
Cadmus performed short-term (two weeks) metering for projects within the nonresidential electric
portfolio. We installed power mete6 and light loggers to obtain operational data to inform energy-
savings estimates. The metering and analysis requirements were specific to the measure category,
Site Visits
Cadmus performed on-site visits to verify measure installations, collect primary data to calculate savings
impactg and interview faciliU staff.
We accomplished three primary tasks during the on-site visits:
1. We verified the implementation status of all measures for which customers received incentives.
We verified that the energy-efficiency measures were installed correctly and still functioned
properly, and also verified the operational characteristics of the installed equipment, such as
temperature setpoints and operating hours.
2. We collected the physical data, such as cooling capacity or horsepower, and analyzed the energy
savings realized from the installed improvements and measures.
3. We interviewed facility personn€l to obtain additional information on the installed system to
supplement data from other sources.
3.2.3. Engineering Analysis
The prescriptive programs and the Site-Specific Program required significantly different methods of
analysis.
Overview
Our procedures for verifying savings through an engineering analysis depended on the type of measure
being analyzed. The following analytical methods were included in this evaluation and are described in
the following sections:
. Prescriptive deemed savings
. Short-term metering
o Billing analysis
80
S.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
Khawaja, The Cadmus Group, lnc
Schedule 1, Page 86 of 130
Calculation spreadsheets
Energy simulation modeling
P rescri ptive De e m ed Soving s
For most prescriptive measures, Cadmus verified the deemed savings estimates Avista used. We focused
ourverification activities on the installed quantity, equipment nameplate data, and operating hours, as
well as on the proper installation of equipment. Where appropriate, we used data from site verification
visits to re-enalyze prescriptive measure savings using Avista's Microsoft Excelo calculation tools,
ENERGY STAR calculation tools, RTF deemed savings, and other secondary sources.
Metering
Depending on the site and measure, Cadmus determined whether short-term metering (over a period of
two weeks) would be most appropriate for achieving precision in that particular project's energy-saving
calculations. Specific metering details for each measure category are discussed in the Results and
Findings section. The installed metering equipment encompassed:
. HoBo light loggers for 12 lighting projects.
Energy Logger Pros for metering two Energy Smart Grocer projects: anti-sweat heater controls
and refrigeration compressors.
Energy Logger Pro for metering fan usage for one site-specific HVAC cooling project.
Energy Logger Pros for metering energy use for seven compressed air and industrial process
motor projects.
Our analysis for each project varied by the measure and metering data obtained.
Billing Analysis
Cadmus analyzed Avista's metered billing data for several site-specific HVAC projects. Using a pre- and
post-modeling approach, we developed retrofit savings estimates for each site. This modeling approach
accounted for differences in HDDs between years. lt also determined savings based on normalized
weather conditions, since the actual weather conditions may have been milder or more extreme than
the TMY3 15-year normal weather averages from 1991-2005 obtained from the NOAA.
We also obtained daily weather data from NOAA for each weather station associated with the
participant projects, then calculated the base 65 reference temperature HDDs. We matched the
participant billing data to the nearest weather station by ZIP code, then matched each monthly billing
period to the associated base 55 HDDs.
We followed a modified PRISM approach for developing the analysis models, which normalized all
dependent and independent variables for the days in each billing period and allowed for model
coefficients to be interpreted as average daily values. We used this methodology to account for
differences in the length of billing periods. For each project, we modeled the ADC in kWh as a function
of some combination of average standing base load, HDDs, and (where appropriate) daily consumption.
a
a
a
81
Exhibit 3
Case Nos. AVU-E-14 AVU-G-'|4
Khawaja, The Cadmus Group, lnc
Schedule 1 , Page 87 of 1 30
S.
For each site, cadmus estimated two demand models: one for the pre-period and one for the post-
period. We chose this methodology over a single standard treatment effects model to account for
structural changes in demand that might have occurred due to retrofits.
Cadmus calculated three scenarios after estimating model coefficients for each site. First, we estimated
a reference load forthe previous 12 billing cycles using the pre-installation period model. This scenario
extrapolated the counterfactual consumption, which is what the consumption would have been in
absent the program. We calculated the energy savings as the difference between the counterfactual
scenario and the actual consumption,
Cadmus then estimated two normalized scenarios: one using the pre-model, and one using the post-
model. We used 15-year TMY3 data in both scenarios as the annual HDD and mean annual values for the
usage data. The difference between these two scenarios represents the long-term expected annual
savings.
Co I cu I ot i o n Sp re o d sh e ets
Avista developed calculation spreadsheets to analyze energy savings for a variety of measures, including
building envelope measures such as ceiling and wall insulation. These calculation spreadsheets require
the input of relevant parameters such as square footage, efficiency value, HVAC system details, and
location details, from which Avista-programmed algorithms estimate energy savings. For each
spreadsheet, we reviewed the input requirements and output estimates and determined ifthe approach
was reasonable,
Energy Simulotion Modeling
Avista determined savings for many site-specific HVAC and site-specific shell projects with energy
simulation modeling, choosing eQuest software because ofthe complex interactions between heating
and cooling loads and the building envelope. Avista provided the original energy simulation models,
which we reviewed to determine the relevant parameters and operating details (such as temperature
setpoints) for the applicable measure. we updated the models as necessary based on our site
verification data.
3.3. Results and Findings
3.3,1. Overview
Cadmus adjusted gross savings estimates based on our evaluated findings. Further details by program
are discussed in the following sections.
For most projects, the documentation was readily available and the measures performed close to
expectations. However, some project files contained excessive documentation, ln certain cases, projects
evolved over time based on participant capital availability and interest level. These project files often
included the different iterations of project development, but did not clearly identify the final reported
82
S.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-l4
Khawaja, The Cadmus Group, lnc
Schedule 1, Page 88 of 130
project energy savings and analysis documentation. Cadmus contacted the participants regarding these
measures, but the lack of clarity sometimes caused them to be confused and dismayed.
3.3.2. Prescriptive
Cadmus evaluated savings for a sample of sites across eight prescriptive programs and the Renewables
Program. Table 51 and Table 52 show our evaluated results by program,
Table 61. Evaluated Results for PY aO7Z-PY 2Ot3 Nonresidential Prescriptive Sample - Combined States
Prescriptive Commercial Clothes
Washer
PresciiptiveEmmercial Windows
and Insulation
erescrifiiw rooaleruice
PrescriptiveGr€en Motors Rewind
Prescriptive Lighting
Prescriptive Motor Controls HVAC
Prescriptive PC Network Controls
Prescriptive Standby Generator
Block Heater
Renewables
97
154
35
4,784
24
3
42
11
5,!52
3
3-
isj
1
1
o
3i
N/A
1,865
11,136
2,254
3,150,101
1,069,O27
21,000
1,376
2,582,336
L,O95,447
0
N/AN/A
1,158
ia,qo
63%
148%
fotal
1,849 1,849
N/4 !/A4,257,233 3,638,646
6L%
ezN
97%
o%
!0004
Nll
85%
8i
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 1, Page 89 of 130
Table 62. Evaluated Results for PY 2013 Nonresidential Prescriptive Sample - ldaho Onlya3
Prescriptive commercial Windows
and lnsulation
iresiri p-tir" iooi servlie
Prescriptive Green Motors
N/AN/AN/A
N/A N/n ryAl
98%2,208lnitiative
PlglcIlqv9llqltllq
Prescriptive Motor Controls HVAC
593 4 1,158,327 665,631
N/A .. -- nrn
o N/A, N/A
o N/A' */o
15
4
s8%
N/a
Prescriptive Standby Generator
Block Heater
Renewables
1
l
1
t'r/A
:
N/A ;
qt!1 6iil -5 1,160,s81 55E,839 _-_ s896
Overall, the prescriptive programs' analysis achieved a level of 90/17 confidence and precision. Cadmus
identified several necessary adjustments to the reported savings for the prescriptive programs. These
calculations often rely on reported equipment and operations data, which may vary from the
parameters identified during on-site verification visits and metering.
Our adjustments decreased savings by 10%. The typical adjustments were to correct equipment
efficiency, fuel type, operating schedules, and/or operating parameters as described below:
o Cadmus used lighting logging and verification data to confirm or adjust operating hours for
lighting projects. These adjustments, in addition to those made based on verified fixture counts,
reduced or increased energy savings by varying amounts.
. Avista implementation staff made a data entry error on one census lighting project. The
calculation workbook listed 645 baseline fixtures listed instead of 54. This data entry error
significantly overestimated baseline consumption, and the resulting realization rate was 3%.
However, Avista paid the correct incentive for the proiect.
For one motor controls HVAC pro.iect, Avista provided incentives for two pump VFDS. One of the
pumps was redundant, as only one is operating at any given time. The realization rate for this
project was 50%.
One food service equipment refrigerator had a larger volume than reported, which increased
savings. The resulting realization rate was 157%.
Cadmus evaluated one PC network controls project. The participant installed the system in 2009
and applied for an incentive in December 2009. The project files show that Avista was still
o'Avista did not install any measures in either the Prescriptive Clothes Washer or PC Network Control programs in
ldaho in 2013. Therefore, we omitted those two programs from the table.
a
s.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-'|4
Khawaja, The Cadmus Group, lnc
Schedule 1, Page 90 of 130
attempting to obtain output reports from the control system to verify savings during 2011 and
2012. The incentive was approved in early 2012. Cadmus contacted the facility in October 2012,
but learned the participant had deactivated the PC network control system. As a result, we did
not assign any savings for this project (a realization rate of 0%).
3.3.3. Energy Smart Grocer
Cadmus performed on-site or metering visits at 25 Energy Smart Grocer Program projects, which
represented a mixture of refrigeration case lighting and refrigeration equipment measures. we
calculated an overall realization rate for all PY 2012 and PY 2013 projects in ldaho and Washington, then
applied the resulting realization rate to the savings for each state. Table 53 lists the number of projects
and reported savings for the two measure types we evaluated. Table 54 shows our evaluated results for
the program by state.
2
6
8
88,535
477,M\
555,976
9
8
t7
24,0L2
972,O20
996,032
11
t4
25
ldaho
Washington
8
r7,a--''-
565,976
996,032
503,604
_]:l,!-ffi
L,515,77O
191
485
-T.tai-676 1,562,008
Overall, the Energy Smart Grocer analysis achieved a level of 90/5 confidence and precision. Cadmus
identified several necessary adjustments to the reported savings for the Energy Smart Grocer Program.
These calculations often rely on reported equipment and operations data, which may vary from the
parameters identified during on-site verification visits and metering.
Table 63. Energy Smart Grocer Program Measure Types and Prorects Evaluated
Table 64, Evaluated Results for Nonresidential Energy Smart Grocer Program Sample
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule '1, Page 91 of 130
Our adjustments decreased savings by 5%. The typical adjustments were to correct equipment
efficiency, operatinS schedules, and/or operating parameters as described below:
At one large site, we found that floating head pressure controls were not enabled on the
medium temperature rack. Energy management system (EMS) data showed that the controls
had not been in operation for at least three weeks, but it could easily have been longer as three
weeks is the limit of the EMS trending history. The reduction in energy savings resulted in a 51%
realization rate.
Cadmus applied a PECI benchmarking work pape# to evaluate savings for several doors added
to medium temperature walk-in cases. The adjustment resulted in a decrease in electricity
savings, for a realization rate of 50%.
Cadmus found variation in actual installed LED case lighting quantities during site visits at two
retail chain stores. The stores installed fewer low output LED case lights and more high output
LED case lights than reported. This increased savings, and the resulting realization rate was
712%.
3.3.4. Site Specific
Cadmus performed site visits at 85 site-specific pro.iects, which represent a variety of measure types.
Cadmus calculated an overall realization rate for all projects in ldaho and Washington, then applied the
resulting realization rate to the savings for each state. Table 65 lists the number of projects and reported
savings for the different measure types we evaluated. Table 66 shows our evaluated results for the
program by state.
Sit+specific HVAC
sita-stecific LightinS
_lte-s1gofic$her
Site-Specific Shell
3,450,865
t49,317
4,708,338
6,766,338
4864,862
6,0s3,416
8,756,94,
6,325,724
30
25
2o
10
8 !7
15.4t 359,772 509,089
!118! I _l9l _- 146ee,310 !5 L,4,645,{1
86
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
Khawaja, The Cadmus Group, lnc
Schedule 1, Page 92 of 130
Table 55. Site-Specific Measure Types and Proiects Evaluated
S.
Table 65. Evaluated Results for Nonresidential Site-Specific Sample
ldaho
Washington
214 .
434 :
Total 648 i 85 2L,645,166
27
58
6,945,856
74,699,3LO
7,40t,9L4
14,024,358
2t,426,272 99%
L07%
95%
Overall, the Site-Specific ProSram achieved a level of 90/10 confidence and precision. Cadmus identified
many adjustments to Site-Specific Program project reported savings. Site-specific projects tend to be
more complex, with energy-savin8s parameters and impacts that are more difficult to estimate. ln
addition, the calculations often rely on participant-supplied building, equipment, and operations data,
which may vary from parameters identified during an on-site verification visit.
ln aggregate, the adjustments noted by Cadmus increased savings by 1.5%, driven primarily by the high
realization rate for lighting projects.
Typical adjustments made to the savings values included corrections to equipment efficiency, operating
schedules, temperature setpoints, and building parameters. Cadmus also identified errors in simulation
models and calculation estimates, which resulted in adjustments. Specific adjustments are identified by
major measure category below.
Site-S pecifi c HVAC Adj ustme nts
. Cadmus determined that Avista overestimated cooling savings for one project. We applied an
equivalent full load hours algorithm supported by RTF analysis. This resulted in lower savings,
for a realization rate of 41Yo.
. Avista adjusted the furnace calculator on one project to calculate heat pump savings, and the
resulting values were too high. The result appears to account for the per-unit consumption
instead of energy savings. Cadmus benchmarked the results against ENERGY STAR, and used the
more conservative value. This led to a 14% realization rate.
cadmus conducted a utility billing analysis on one small heat pump project, which revealed no
electricity savings resulting from the project and resulted in a realization rate of 0%.
The heating load appeared to have been overestimated on two large, partially-occupied,
multifamily new construction projects. The utility billing data showed an average 65% of
expected consumption when normalized to full occupancy.
Cadmus engineers found issues with simulation modeling by one contractor on four projects.
The models had an excessive portion of simulation hours outside of the throttling range. The
unmet load hours outside the throttlinS range indicate zones in the model, which do not receive
sufficient heating or cooling. This value should be less than 5% (as recommended by the U.S.
Green Building Council's Leadership in Energy and Environmental Design). Larger values call the
87
S.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
Khawaja, The Cadmus Group, lnc
Schedule '1, Page 93 of 130
integrity of the model into question. These four evaluated projects had unmet load hour issues
ranging from 10.35% to 99.9% for any system zone outside throttling range. However, the
contractor had calibrated the models to the utility billing data. Overall, the energy savings and
model energy consumption appeared to be within a reasonable range. An example ofthe issue
from an eQuest simulation output file is shown in Figure 15.
Figure 16, eQuest Output File Showing Throttling Range lssue
EE- aE Blq btE h!!@ r@ tEl- lpr - rc
ilI EA=g "=I -ro
gE OE
=g:=
9.2 ta1,a
6S-a 0.0
se6.t 12!.4
u l@t:= i-r1
0.0 $-6
0.0 o.o
o.o s.a
EE t@C tttr ffi Bns r@ t@E M@ OSr M[
E !4@EBEA 1152.2
E IEu&4EE O.O
Et 1t82.2
0.0 s3.t
o.o 0.0
0.0 s3.1
1€ 6.a
0.0
1qa.a
o-0 0-0 0.0 0.0 s2r.2
o.o 0.0 1aa.a 0.0 ga*.l
0.0 0.0 tBa.a o.o 9*.3
DEtEtlIlE ls.a?lE 3.oEE/rgr-E@&8A ta-o @68-nE{EDil&EEd t6.atl@ t[.rE!/a9E-4cs]&[ lt1.r@/3gE-nlE.lq
EtsEotm &EE z8 @aEloaC@ M- la.tBuot@Bru@Er@ - 0.0
DE: gE E Dm& EE rc E EB firlE.
Site-Specific Lig hti ng Adjustments
Cadmus evaluated a non-census sample of site-specific lighting projects using a combination of light
logging and verification data. On average, the results indicated reasonable reported values, and the
measure category had a realization rate of 98%.
o Cadmus evaluated the largest project (with 2,857,210 kwh of reported savings) through
extensive verification and Iight logging. The evaluated results were nearly identical to Avista's
reported values, resulting in a 100.5% realization rate.
. On one hotel project, Avista assumed 25 operating hours per week for wall sconces. Light
logging revealed that the fixtures were never turned off This increased the baseline and retrofit
energy consumption. Thereforg it also increased energy savings, resulting in a 306% realization
rate.
o On one small new construction project, the installed lighting power density exceeded code
requirements; therefore, no savinSs could be achieved and the realization rate was 0%.
88
S.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
Khawaja, The Cadmus Group, lnc
Schedule 1, Page 94 of 130
Site -S pecif ic Oth e r Adj ustme nts
o Cadmus found that Avista applied an incorrect baseline for a refrigerated dryer on a compressed
air application. The baseline listed a desiccant dryer, which would actually consume far more
energy than Avista estimated. The refrigerated dryer is the industry standard, and typically
represents the baseline. Thus, no savings were achieved for this project.
. Cadmus metered two industrial process motor projects and one compressed air project, and
accepted Avista's metering data for baseline energy consumption. Our metering data indicated
lower retrofit energy consumption than Avista's retrofit data. This would increase energy
savings. We compared the production data for both periods, and could not reconcile the
difference in energy consumptlon based on that data. We therefore combined the Avista and
Cadmus retrofit metering data to establish the normalized retrofit energy consumption. The
realization rate for these three projects was 85%.
o Cadmus adjusted savings for a small refrigeration circulation pump project to match actual
operating hours. This resulted in a reduction in energy savings, with a realization rate of 33%.
o Cadmus evaluated the remaining site-specific other projects using a combination of utility billing
and verification data, On average, the results indicated that the achieved energy savings were
slightly less than the reported values.
Site-S pecilic Sh e ll Adj ustme nts
. One site-specific shell project had low evaluated savings based on the initial calculation
methods. Avista funded the switch from electric resistance to natural gas heatin& but did not
update the shell calculator with new fuel, and calculated shell savings in terms of electricity. The
resulting realization rate wes 35%.
. Cadmus performed a site visit at one school with two site-specific shell projects. We found that
the site turned off their HVAC system completely during the summer months when school was
not in session. Avista based its energy-savings estimate on the assumption that air conditioning
would operate during the summer months. This required an adjustment to reduced energy
savings, with a resulting realization rate of 34% for both projects combined.
Cadmus evaluated the remaining site-specific shell projects using verification data with the applicable
Avista savings calculators. ln general, Cadmus found that the reported shell quantities and properties
did not vary much from verified values, and the savings calculators produced reasonable results. The
remaining results indicated that the achieved energy savings were equal to the reported values.
3.3.5. Extrapolation to Program Population
For our evaluation of the nonresidential electric programs, we selected sites that could provide the most
impactful information. We designed the site visits to achieve a statistically valid sample for the major
strata, as discussed previously. For measures in the random (non-census) sample, we calculated
realization rates (the ratio of claimed-to-verified savings) and applied these to the remaining non-
sampled sites. We did not apply measure-level realization rates to the census population. These
89
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
Khawaja, The Cadmus Group, lnc
Schedule 'l , Page 95 of 130
realization rates are weighted averages, based on the random verification sample and using the
following four equations.
We calculated realization rates for each individual site in the sample based on measure type:
Verified..
RR,, = ,, " ,v :,formeasurejatsitei' Claimed u ''
Where:
RR = Realization rate
i = Sample site
j = Measure tYPe
Then we calculated the realization rates for the measure types using the ratio of the sum of verified
savings to the sum of claimed savinSs from the randomly selected sample for each measure type:
Itterilel ,
RR, = a.
-
;formeasure j acrossall samplesites
Lclarme.lti
We calculated the population-verified savings for non-census projects by multiplying the measure type
realization rate from the random sample by the claimed savings for the non-census population of each
measure type:
lrerya r = RR jx>Claimed r; for musure j across all sita in measure populatbnkt
Where:
k = Total population for measure type j
Finally we added the claimed and verified savings from census stratum measures to calculate the total
reported and verified savings for each program. The program realization rate is the ratio of all verified to
all claimed savings:
).uerilea roo - k
-;
for the population (all sites and measures),",,_lctor
"ark
Where:
| = Total program population
Cadmus summed these values to determine the total adjusted evaluated savings and program-level
realization rates for the programs as a whole and for ldaho and Washington, as shown in Table 67 and
Table 58. The overall portfolio gross realization rate was 97%.
90
S.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-l4
Khawaja, The Cadmus Group, lnc
Schedule 1, Page 96 of 130
E@
Table 67. PY 2OLZ-PY 2013 Electric Gross Program Realization Rates - Combined States
Prescriptive 4,257,233 3,638,546
1.5t5.770
95% 6,791,118
92% 22,560,559
5,229,048 9L% 3,367,s37
9,141,338 t70% 9,596,933
Site-Specific shell 509,089
Total 27,464,407 25,580,588 97% 47,091,646l Realization rates vary from the ratio of evaluated to reported savings due to the impact of census-level prcrects.
5,559,011
396,875
Prescriptive 1,150,581
Energy Smart Grocer 449,M3
Site-Specific HVAC 759,054
Site-specific Lighting L,U2,534
Site-Specific Other 2,38L,238
Site-Specific Shell 113,857
6,706,707
* Realization rates vary from the ratio of evaluated to reported savings due to the impact of census-level proiects.
3,4. NonresidentiolConclusions
Cadmus evaluated f42 of 6,476 measures installed through the nonresidential programs, representing
15% of reported savings.
ln general, Cadmus determined that Avista implemented the programs well. The overall portfolio
achieved a 97Yo realization rate when comparing gross evaluated savings to gross reported savings. ln
ldaho, the PY 2013 nonresidential portfolio achieved a 94% realization rate.
Cadmus identified the following key issues that led to adjusted energy savings:
. Metering on post-installation power consumption for several industrial process measures
indicated that the evaluated savings varied from the reported value.
. Some participants did not operate the incented equipment correctly or did not complete the
improvements expected for the measure.
o Some participant post-installation heating or cooling loads did not achieve the level of projected
consumption, which reduced energy savings.
Energy Smart Grocer
site-speciiii nvnc
Site-Specific LiBhting
Site-Specific Other
Total
558,839
397,974
666,597
2,280,5L8
2,775,69L
5r,cii
6,257,446
L@% 4,693,462
78% 82,037
86% 8,O79,rO795% r,zss,goa
89% L,I04,O62
Lr3% 3,483,430
96% 3,LLL,7!8
78% 70,108
94% t7,602,25r
5,448,089
20,652,917
3,053,019
10,589,164
4,696,253
53,954
45,s03,4s6
6,978,966
L,672,\3g
977,838
3,gLg,2gg
2,9g2,M5
s+,oss
15,595 342
1,562,008
6,053,406
8,756,943
6,325,72A
Table 68. PY 2013 Electric Gross Program Realization Rates - ldaho Only
97
S.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
Khawaja, The Cadmus Group, lnc
Schedule '1, Page 97 of 130
. Simulation models sometimes did not accurately represent the actual as-built building or system
operation.
o There were instances where thorough analysis of energy-savings calculations provided by
participants or third-party contractors was lacking.
. Some projects had data entry errors in characterizing building or measure performance.
3.5. Nonresidential Recommendotions
Cadmus recommends that Avista continue to offer incentives for measure installation through the
evaluated programs. We have the following recommendations for improving program energy-savings
impacts and evaluation effectiveness:
. Create a quality control system to double<heck all projects with savings over 300,000 kwh.
. Avista may want to consider tracking and reporting demand reduction to better understand
measure load profiles and peak demand reduction opportunities.
. Update prescriptive measure assumptions and sources on a regular basis.
. Streamline its file structure to enable reviewers to more easily identiry the latest
documentation.
. Continue to perform follow-up measure confirmation and/or site visits on a random sample of
projects (at least 10%).
o Consider flaeBing sites for additional scrutiny when the paid invoice does not include installation
labor as it may indicate that the work was not yet performed.
. Avista may consider adding a flag to their tracking database to automatically calculate the unit
of energy savings per dollar (kWh/S or therm/$) to provide a quick check to identiry extreme
outliers.
o ln the case of redundant equipment, Avista may want to consider incenting pump projects
through the Site-Specific Program to more accurately characterize the equipment operating
hours.
. Avista may want to set minimum standards for modeling design guidelines. The Energy Trust of
Oregon provides an example on their website:
htto://enerqvtrust.orqlcommercial/incentives/construction-renovation-
improvements/custom/modeled-savinss.
92
Exhibit 3
Case Nos. AVU-E-'|4 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 1, Page 98 of 130
4. Low lncome lmpact Evaluation
4.7. lntroduction
Cadmus conducted a statistical billing analysis to determine ad.iusted gross savings and realization rates
for energy-efficient measures installed through the low-income weatherization program in PY 2013.
Cadmus examined energy savings at the household or participant level, rather than at the measure level.
We performed a billing analysis of PY ?OLZ participants who had a full year of energy consumption data
both before (2011) and after (2013) the weatherization period. Then Cadmus applied PY 2012 billlng
analysis results to PY 2013 participants.
To estimate energy savings resulting from the program, Cadmus used a pre- and post-installation,
combined CSA and PRISM approach, using monthly billing data. We analyzed energy-savings estimates
for program participants and ran a series of diagnostic tests on the data, These tests included reviewing
savings by pre-consumption usage quartile, checking to ensure households have a sufficient amount of
billing data, and creating a graphical outlier analysis. Below is a detailed discussion of the regression
model used for this billing analysis along with resulting savings.
4.1.1. Program Description
Five components, listed in Table 69, are included in the low-income weatherization program. Local
Community Action Partners (CAPS) within Avista's ldaho service territory implement the projects. CAPS
holistically evaluate homes for energy-efficiency measure applicability, combining funding from different
utility and statefederal programs to apply appropriate measures to a home, based on the results of a
home energy audit.
Table 59. Low-lncome Weatherization: PY 2013 Electric-Efficiency lnstallations by Component*
ShellAiVeatherization lnsulation, window/door, air infiltration, programmable thermostat
Fuel Conversion*rteciiic iumace, heai pump, or wateitreater repiacemen-
Hot Water Efficiency High€fficiency water heater replacement
ENERGYSTARAppliance High;fficiencyrefriSeratorreplacement
HVAC Efficiency High-efficiency heat pump replacement, variable speed motor
* Avista considers (and reports) fuel conversion measures in its portfolio as electric-saving measures.
93
270
35
0
o
2
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
Khawaja, The Cadmus Group, lnc
Schedule 1, Page 99 of 130
S.
4.2. Doto Colledion ond Methodology
Cadmus obtained impact evaluation data from multiple sources, including:
. Progmm pottlclwnt dotobose: Avista provided information regarding program participants and
installed measures. Specifically, these data included a list of measures installed per home and
the reported savings from each completed installation. The data did not, however, include the
quantity of measures installed (such as the total square feet of installed insulation) or per-unit
savings estimates.
o Billing records: Avista provided participant meter records from January 2011 through December
2013.
. Weathet doto: Cadmus collected ldaho weather data from NOAA for three representative
stations, drawn for the corresponding time period.
4.2.1. Sampling
Cadmus began the analysis with a census of PY 2012 participants. We then screened the PY 2012
participant data for specific criteria (e.g., ensuring that it had sufficient monthly billing data, was not
classified as an outlier) for use in the final analysis. ln all, we included 65 ldaho participants in the billing
analysis: 50 non-conversion and 15 conversion participants, Cadmus defined a conversion customer as
any participant who received a new gas furnace or water heater.
4.2.2. BillingAnalysis
Avista provided monthly billing data for all participants from January 2011 through December 2013.
Avista also provided the participant database, which contained participation and measure data forthe
PY 2012 and PY 2013, detailing all gas and electric measures installed per home by CAPs.
Cadmus obtained daily average temperature weather data from 20U to 2013 for the three NOAA
weather stations, representing all PY 2012 electric participant ZIP codes in Avista's ldaho territory. From
daily temperatures, we determined base 65degree HDDs and CODS for each station, then matched
billing data periods with the HDDs and CDDs from the station closest to each participant.
As we received billinS data through December 2013, we could only perform the billing analysis for the
2012 program year. We defined the analysis pre-period as 2011, before all participation installations
occurred, and defined the analysis post-period as 2013, following all installations occurring in 2012. We
then applied the analysis results for PY 2012 participants to the PY 2013 participant population, thus
reporting overall impacts for PY 2013. Given consistency in delivery infrastructure, measure offerings,
and program design, using billing analysis and extrapolating evaluated impacts from the previous year to
2013 seems appropriate. Furthermore, performing billing analysis for whole-house programs is
considered an industry best-practice, cited in several evaluation protocols (lPMVP, UMP), allowing for
the utility to account for measure interaction, participant take-back, and the effects of energy-education
on participant usage behavior.
94
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
Khawaja, The Cadmus Group, lnc
Schedule 1, Page 100 of 130
S.
To estimate energy savings from this program, Cadmus used a pre/post CSA fixed-effects modeling
method using pooled monthly time-series (panel) billing data. This modeling approach corrected for
differences between pre- and post-installation weather conditiont as well as for differences in usage
consumption between participants (as the model included a separate intercept for each participant).
The modeling approach ensured that model savings estimates would not be skewed by unusually high-
usage or low-usage participants.
4.3. Dota Screening dnd Modeling Approqch
Cadmus conducted a series of steps to screen participant usage data, ensuring a clean, reliable dataset
for analysis.
4.3.1. General Screens
Cadmus used the following screens to remove accounts that could have skewed the savings estimation:
. Accounts with fewer than three months (90 days) of billing data, in either the pre- or post-
period.
. Accounts with annual usage outside of reasonable bounds in either the pre- or post-period (less
than 1,000 kwh or more than 50,000 kwh).
. Accounts that change electric usage between the pre- or post-period by more than 90% (unless
for a conversion project).as
4.3.2. WeatherNormalizationScreens
To screen the data, Cadmus used PRISM-like models to weather-normalize pre- and post-billing data for
each accoun! and to provide an alternate check on measure $vings obtained from the CSA model. For
more detail on the model specification, see Appendix E.
Table 70 and Table 71 summarize non-conversion and conversion account attrition, respectively, from
the screens listed above.
Changes in usage ofthis magnitude are probably due to vacancies, home remodeling or addition, seasonal
occupation, or fuel switching. Changes of usage over a certain threshold are not expected to be attributed to
program effects and can confound the analysis of consumption.
95
S.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
Khawaja, The Cadmus Group, lnc
Schedule 1, Page 101 of 130
Table 70, Low-lncome Weatherization: Non-Conversion Account Attrition
Drgppedi! M.ege lvith Billins Data
lnsufficient Pre- and Post-Period Months
lnsufficient Pre- and Post-Days
Low or High Usage in Pre- or Post-Period
CfingeO ur"g" setwen Pre and Post (>
9O%l
PRISM Screen: Low R-Squared, Low HeatinB
Usage
97% ,
95%
gs%
63
51
61 ;
50
5o
r00%
0
1
0
0%
2%
o%
g%
2%
0
I
60
59
L4%
L'
50
591I
o%
o%
o%
95%
g4%
94%o%59
Account-level inspection of prelpost 12-
month usage (e.g., vacancies, anomalies)
Final Anafysis Group
Original Electric Accounts
Dropped in Merge with Billing Data
Lniutriciintp[- inJ ioiiperioa Monttrl
lnsufficient Pre- and Post-Days
Low or High Usage in Pre- or Post-Period
79%
79%
100,6
100%
zL%
18
18
18
18
i8
0
o
o
too% t:
100%.
L0Oor6
Changed Usage Between Pre and Post (>
9O%) 18 1Oo%
PRISM Screen: Low R-squared, Low Heating oo%
" -.;:_
18 100% o o%Usage
.r1583%317%
Tol-tl1ll"gj (":errrc?nci::r 3nomllie1 i ': r !'h
FinalAnalysiscroup 15 A3% ! ll7%
4.3.3. Conditional Savings Analysis Modeling Approach
To estimate energy savings from this program, Cadmus used a pre/post CSA fixed-effects modeling
method, which uses pooled monthly time-series (panel) billing data. The flxed-effects modeling
approach corrects for differences between pre- and post-installation weather conditions, as well as for
differences in usage consumption between participants with a separate intercept for each participant.
This modeling approach ensured that model savings estimates are not skewed by unusually high-usage
or low-usage participants. For more detail on the model specification, see Appendix E.
96
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 1 , Page 102 of 130
Table 71. Low-lncome Weatherization: Conversion Account Attrition
4.4. Results and Findings
This section presents the evaluated savings for the program derived from the billing analysis. Several
detailed tables are presented to contextualize the billing analysis impacts, including measure
distributions and some benchmarking comparisons.
4.4.1. Billing Analysis Results
Table 72 summarizes model savings results forelectric non-conversion and conversion participants of
the low-income weatherization program.
1er9,?8
16,859 10,980
L5% r30%
65% t19%
L,943
8,890
3,609
13,O7L
The model savings averaged 2,775 kWh for each non-conversion participant and 10,980 kWh for each
conversion participant. ln this analysis, Cadmus determined an overall conversion estimate instead of
equipment-specific estimates due to the small sample size of furnace-only and water heater-only
participants at the state level.
Table 73 provides a distribution of the electric measures in the final model that Avista funded for
participants. This distribution reveals a different mix of measures for the two participant groups.
Specifically, non-conversion participants had higher installation percentages of shell measures (e.g.,
doors, windows, wall insulation).
97
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
Khawaja, The Cadmus Group, lnc
Schedule 1 , Page 'l 03 of 130
Table 72. Electric Model Savings Summary
S,
Table 73. Measure Distribution of Final Model Sample by Participant Type
7%
7%
Floor insulation
Atticinsulation -
Or.t n.utation '.
Water heatir replacement
Wall lnsulation
T-stat (no ;ir conditioning)
Refriger_ator replacement
.Furnace replacement
Furnace conversion
18 36% I 7%
L4 28% O O%
36%0W6 1..._11 22% L 7%
13 26% O O"fr
t2%oo%
9. W',. 7 47%
0: Vn O O%
o o% \4 93%
0 o%l L4 93%- -::- ''o oe6 i 13 87%50- too% t5--- 1m9g
9r%
367%
Water heater conversion
sample (nl
Statistical billing analysis resuhs encompass all measure installations made at participant households,
including those not paid for through Avista's program. Since local CAP agencies use a variety of funding
sources to implement this program, it is possible that participant homes received measures paid for by
federal, state, and/or other utility dollars. Specifically, Avista does not fund CFLs offered through the
program, which likely had a significant impact on the electric savings in participant homes.
4.4,2. Overall Program Results
Table 74 shows the realization rates for ldaho low-income weatherization program participants.
Table 74. Low-income weatherization: Electric Model Realization Rate 5ummary
2,776
rc,480
t6%
iC%
Non-conversion participants had a realization rate of 91%. There were two PY 2013 participants who
received electric resistance to electric heat pump conversions, which were not represented in the billing
analysis sample.
Cadmus used Avista's listed database savings for the heat pump conversion measures and additional
non-conversion measures for this customer. Table 75 presents the PY 2013 population savings separated
by participant type.
98
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 1, Page 104 of 130
E@
Table 75. Low-lncome Weatherization: Total PY 2013 Evaluated Savings
Non-Conversion
converiion
Heat Pump Replacement*
Overall 499,902 292,767
* Avista funded high-efficiency electriJheat pump replace.enti that *ere not included in the billing analysis
participant sample. For these measuret Cadmus used the claimed savings values listed in the Avista database.
Cadmus calculated the total program savings by multiplying the modeled realization rates by the
claimed ex onte savings.
4.5. Comparison to Previous Billing Anolysis
The results from the PY 2012 billing analysis indicate greater energy savings than had resulted from the
PY 2010 billing analysis. Table 75 compares the model results from Cadmus' PY 2010 and PY 2012 billing
analyses.6
Table 76. Low-lncome Weatherization: Comparison of Model Results by Participant Group and Year
L79,628 197,945
23 10,980 309,954 84,513
2 N/A 10,309 10,309
100 2,776
125 N/A
Non- 2010 73 15,773 t,6OZ 3,626
Conversion 2ot2 50 is,o98 2,776 3,059
* The model results are not statistically different at the 0.05 level of significance
One factor contributing to increased modeled energy savings between PY 2010 and PY 2012 is a change
in the distribution of electric-saving measures that Avista funded. Avista funded a greater number of
high energy-saving measures in PY 2012 than in PY 2010 for non-conversion participants, including air
infiltration controls, floor and duct insulation, and doors. Additionally, Avista began funding wall
insulation and water heater replacements in ldaho. Figure 17 shows the percentage of Avista-funded
measures for non-conversion perticipants for both program years.
No comparison is provided for fuel conversion measures, as Avista added these measures to the ldaho
program after the previous evaluation.
99
9t%
g67%
N/A
rztx
44%
9L%
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 1, Page 105 of 130
Air infiltration controls
Attic insulation
Doors
Duct insulation
Floor insulation
Wall insulation
Water heater replacement
Windows
)PY 2OL2
. PY 2010
The realization rates are also substantially higher in PY 2012 than in previous years. As explained above,
there was an increase in the installation of building shell measures during PY 2012. The difference in
realization rates is also partially due to the reported measurelevel savings. Table 77 presents a
comparison of the average kwh savings between PY 2011 and PY 2012-PY 2013 for both non-conversion
and conversion customers.
Table 77. Comparison of Average Reported Measure-Level Savings Between Program Years*
Air infiltmtion controls
ASHP replacement iconversion)
Attic jnsulalion
Doors
Duct insulation
Floor insulation
t,87L 458 ;
3,rrt I
s89 i
313 i
1417
L,874
N/A
L,478
287
5,485
4,40c
Furnace replacement (conversion)
High€fficiencywater heater replacement
N/A 1s!LLt7
L,O75 )
1,148 I
1,255 ;
299
Wall insulation 3,466
Water heater replacement (convereionJ N/A
Windows 2,432
* These savings values reflect full program yeas, not the analysis sample.
All but one measure (doors) experienced a decrease in average reported savings between PY 2011 and
PY 2012-PY 2013. The measures with the largest change in reported savings were air infiltration, attic
insulation, wall insulation, duct insulation, and floor insulation.
7N
S.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
Khawaja, The Cadmus Group, lnc
Schedule '1, Page 106 of 130
An additional factor may account for changes in modeled savings: non-Avista funded measures installed
by agencies through the program.
4,6. Benchmarking
To place Avista program savings estimates in context, we compared billing analysis results from other
low-income program efforts across the country.o7 This section provides two metrics for comparing
Avista's program savings to other similar programs. First, Figure 18 compares the percentage of energy
savings, relative to PRENAC, of Avista's program and a number of other low-income weatherization
programs, based on electric billing analyses, This metric allows for comparing programs given variation
in weather, costs, program delivery and measure offerings.
Figure 18. Savings Percentage of Pre-Period Consumption*
Figure 19 presents the absolute energy savings from low-income programs; this is a second metric for
comparing Avista's non-conversion results to other programs. Absolute estimates do not use PRENAC,
but rather show savings that are directly attributable to the program.
The comparable studies include oak Ridge National Laboratovs (oRNL) Meta-evaluation of Low-lncome
weatherization Programs, the People Working Cooperatively Low-lncome Weatherization Program in Ohio
(MW), the Pacific Power (PP) Low-lncome Weatherization Program in Washington, the Rocky Mountain Power
(RMP) tow{ncome Weatherization Program in ldaho, the Energy Smart low-income program in Or€gon (OR).
101
Avista - lD (PY12-13)
Avista - lD (PY10-11)
ORNL (metaeval.)
Mw Utility (2009)
OR Energy Smart (2007)
RMP rD (07-09)
PP WA (09-11)
o% 2% 4% 6% a% LO% t2% t4%
Percent kwh of PRENAC
* This figure reflects savings for non-conversion participants.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
Khawaja, The Cadmus Group, lnc
Schedule 1, Page 107 of 130
S.
Figure 19. Average Per-Participant Savings for Non-Conversion Participants
Avista - lD (PY12-13)
Avista - lD (PY1&11)
oRNL (metaeval.)
MW Utility (2009)
oR Energy Smart (2007)
RMP rD (07-09)
PP WA (09-11)
500 1,000 1,500 2,000 2,500 3,000
kwh Savings per Participant
The realization for ldaho conversion participants appeared high, at 357%. When comparing PY 2013
average savings for furnace and water heater conversions in Avista's tracking data, average savings for
ldaho were 50% of the Washington estimates for these measures. By comparison, billing analysis results
for Washington conversion participants over the last two studies are consistent with the modeled
savings here for ldaho, coming in at 8,394 kwh and 10,397 kWh, respectively.
4.7. Low-lncome Conclusions
Compared to PY 2010, Avista's PY 2012 low-income program demonstrated an increase in average
electric savings per participant, in addition to an increase in non-conve6ion program realization rate
(frcm 44%to 9L%). Several factors may have contributed to the increase in non-conversion participant
savings, including: (1) an increased frequency of installing high-saving measures (e.g., shell) in the
evaluation period, (2) chan8es in agency delivery protocols or energy-saving installations made with
non-utility fundin& and (3) exogenous effect (e.9., economic, rate changes) that may have occurred
simultaneous to program activity. One factor contributing to higher realization rates are lower average
reported savings occurring in the evaluation period compared to previous years.
While we cannot compare the results of the conversion customer impacts to previous evaluations in
ldaho, average savings are comparable to those observed in the Washington low-income program
through past billing analyses.
102
S.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
Khawaja, The Cadmus Group, lnc
Schedule 1, Page 't08 of 130
4.8. Low-lncomeRecommendations
Cadmus recommends the following enhancements in order to improve program impact results:
o Avista should use a control or comparison group in future billing analyses for use in analyzing
the treatment group of program participants. This would allow controlling for exogenous factors
(e.g., macroeconomic, rate changes, technological trends) that could result in trends that affect
consumption. Controlling for these trends using a control/comparison group is a robust and
defensible method for estimating accurate energy-savinSs impacts.
. Avista should consider options for increasing analysis sample sizes (such as using combined
models with participants in either state program). Smaller sample sizes in state-specific models
attributed to decreased precision in the PY 2012 model estimates. lncreasing the sample sizes
by using a combined state model in future evaluations will mitigate this cause of decreased
precision.
. Avista should obtain a full list of weatherization measures from agencies. The billing analysis
results do not allow Cadmus to disaggregate energy savings specific to Avista-funded measures.
ln addition, a complete list of participants' installed measures would allow Cadmus to conduct a
measure-level billing analysis specific to measure types. This granularity could help Avista
improve future program offerings and help fully characterize the energy savings modeled
through billing analysis.
. Avista should include high-use customers in program targeting. While prioritization guidelines
for targeting low-income weatherization participants are set at the federal level, some utilities,
for targeting purposes, actively track customer usage and provide agencies with lists of
customers that have particularly high energy consumption.
Notably, DOE protocols list high-energy consumption as a factor allowed in participant
prioritization. ln such cases, along with other targeting criteria (e.9., families with children,
senior citizens), agencies may incorporate energy-consumption characteristics into their
program participant prioritization. Not only would weatherizing high-use customers likely result
in higher energy savings, but could provide these customers with some financial relief from their
higher energy bills caused by their housing characteristics.
Avista should identiry high-usage customers while controlling for factors that contribute to
consumption (e.g., square footage, income, numbers of people per household).
Given reductions in federal funding for weatherization and associated reduced agency capacities
resulting in more limited leveraging opportunities, Avista can lead new efforts for the continued
delivery of energy-savings resources to low-income residential customers. Potential exists to
secure cost-effective energy savings through high-usage targetinE, while continuing to support
weatherization for income-qualified customers. Efficient targeting balances efforts to provide
whole-house weatherization, and allows for levera8ing the agency network as a resource for
outreach and delivery.
103
S.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
Khawaja, The Cadmus Group, lnc
Schedule 1, Page 109 of 130
Avista should track and compile additional data from agency audits. These data include
information on primary and secondary heating and coolin& and on the size of a home. As an
inexpensive alternative to gas heat, gas customers may turn to electric room heaters and wood
stoves, reducing the impacts of installed weather-sensitive measures (e.9., insulation), Collecting
information on customers'primary heating usage during weatherization would lead to more
reasonable savings estimates.
Cadmus recommends that Avista work with CAP agencies to develop explicit, on-site tracking
protocols for collecting information on participant heating sources. The CAPs should collect the
following information to better inform heating and cooling sources:
. Visual inspections of all heating equipment found on site;
. Participant-reported primary and supplemental heating sources used;
. Quantities of secondary heating, if applicable (e,9., numbers of electric room heaters); and
r Any indicators suggesting discrepancies between actual and reported primary heating,
Avista should consider performing quantitative, non-energy benefit analyses. Cadmus
recommends that Avista consider pursuing additional analyses aimed at quantirying non-energy
benefits associated with low-income weatherization, applicable to the Total Resource Cost (TRC)
test. Specifically, analyses of economic impacts and payment pattern improvements (including
reduced arrearages and collections costs) can provide program stakeholders with the monetized
value of energy-efficienry measures. Other Northwest utilities have used such analyses to report
low-income weatherization cost-effectiveness (in ldaho and Washington). Standard cost-
effectiveness TRC testing accounts for all program costs and only includes energy savings as a
program benefit. The TRC test omits some non-energy benefits genuinely experienced by
participants, such as decreased mortality and morbidity, as well as environmental benefits such
as reduced emissions of carbon dioxide and other pollutants listed in the Clean Air Act.
1A
Exhibit 3
Case Nos. AW-E-14 AVU-G-l4
S. Khawaja, The Cadmus Group, lnc
Schedule 1, Page 110 of 130
5. Portfolio Savings and Goals
5.7. Gross Portfolio Sovings
The PY 2013 ldaho electric portfolio consisted of several sectors and many program delivery streams. ln
total, the programs achieved a 7O2,7Yo gross realization rate and total evaluated savings of
2s,899,345 kWh (Table 78).
Residential
Nonresidential
Low lncome
Residential Behavior
5,130,507
17,602,253
292,767
2,L94,322
5,933,197
t6,595,342
499,901
2,870,905
115.6%
94.3%
tto.sx
rEo.ax
ro2.7%Total 25,219,U9 25,899,345
* Note that Residential Behavior Program savings are inherently calculated as net, not gross,
5.2. NTG Adjustment
Cadmus evaluated NTG through customer self-reports, using different methodologies and data sources
for the different programs, as detailed below.
5.2.1. No NTG Adjustment
The programs outlined below did not require a NTG adjustment, as the original savings analysis
methodology accurately reflected net market characteristics.
Low lncome Weotherizotion
Traditionally, low-income programs receive a 100% NTG as the participants are assumed unlikely to have
installed the incented measures on their own.
Simple Steps, Smort Sdvings and Geographic CFL Giveaway
The savings analysis methodology Cadmus used for Avista's upstream and giveaway lighting programs
follows the RTF, which does not differentiate between gross and net savings but instead uses an
adjusted market baseline approach. Forthe various inputs to the savings calculation, Cadmui used
either direct RTF values or RTF methods with Avista-specific data. To assign an additional NTG value to
these programs would, in effect, be double counting.
Residentiol Behavior
Cadmus analyzed the Residential Behavior Program using a randomly selected control group such that
the differences between groups net out any natural effect of what people would have done in absence
of the program, or because of the existence of the other Avista programs. The savings produced by this
method of analysis are inherently net and need no further adjustment.
105
Table 78. PY 2013 ldaho Gross Savings
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule '1, Page 111 of 130
5.2.2. Residential NTG
Cadmus updated NTG values for the PY 2013 residential population. We determined freeridership and
participant spillover from 210 participating customers' self-reports during phone surveys performed in
Q1 2014. The methodology is consistent with that described in detail in Cadmus'2012 NTG report.*
We calculated nonparticipant spillover from 1,109 completed multi-method General Population surveys
(395 of which were ldaho residents). We mailed 3,000 paper surveys to randomly selected residential
customers in both ldaho and Washington. These mailings included a website to complete the survey
online. Cadmus also called a subset of the sample with a traditional phone survey. This multi-media
method helps reduce survey bias.
Cadmus followed a specific NTG methodology for the Second Refrigerator and Freezer Rerycling
Program, as outlined in the program section above. Table 79 outlines the NTG components and resulting
program-level NTG from our most recent round of analyses.
ENERGY STAR PToducts
xeitiru "no coorirutmcilncy
Weatherization/shell
72%__.]_
55%
46%
39%
0.0%
oo%
oo%
o-7%
o.7%
29%
46%
Water Heater Efficiency s-sr-l 9t!)
o.7%Space and Water Conversions 62%0.096
Table 80 shows the NTG values and resulting net savings for Avista's residential downstream programs
(36%), and a NTG for the residential sector ove'all (92%1.
{ Cadmus. Net-to -Gross Evoluotion of Avisto's Demand-Side Monogement Progroms. June 2OLz
1M
Table 79. Residential NTG
Exhibit 3
Case Nos. AVU-E-14 AVU-G-'|4
S. Khawaja, The Cadmus Group, lnc
Schedule '1, Page 112 ot 130
Table 80. Residential NTG and Net Savings
Second Refrigerator and Freezer Recycling
ENERGY STAR Products
Heating and Cooling Efficiency
Weatherization/Shell
Water Heater Efficiency
Space and Water Conversions
ENERGY STAR Homesr
subtotal
Simple Steps, Smart Savings
Geographic CFL Giveaway
Residential Behavior
Total
368,L74
29,OLL
L44,480
90,47L
s,nai
506,078
12,550
1,156,251
4,750,306
25,640
2.870.905
8,804,102
32%
23%
29%
45%
46%
39%
7qx
36%
tw%
roo%
700%
tL7,699
6,760
42,L88
4L,496
2,513
195,345
9,287
irs,zig
4,750,306
26,640
2,870,905
8,063,080
*ENERGYSTARHomesNTGwasnotevaluatedin2013duetosmallparticipation. Valueisfrompreviousevaluation.
5.2.3. Nonresidential NTG
Cadmus surveyed PY 2013 participants in Q1 2014, following the methodology described in Cadmus'
2012 NTG report. Table 81 outlines the NTG components and resulting programJevel NTG.
Site-Specific
Prescriptive
,Energysmart Grocer
I Site-Specific
I Prescriptive
I Energysmart Grocer
i Total
30.4%
g.r%
L4.3%
o.t%
o.o%
o.o%
08%
o.8%
o.a%
Table 82 shows the resulting net savings for each program component. The nonresidential sector
exhibited a weighted nonresidential NTG of 81%.
7,944,237
5,978,956
7,672,139
16,595,342
70.4%
grI%
86.5%
8t%:,
5,594,332
5,396,222 :
7,445,564 :
13,436,118 l
5.3 Net Portfolio Savings
The portfolio achieved an overall NTG ratio of 85% and 21,999,099 kWh of net savings. Table 83 shows
evaluated gross and resulting net savings for ldaho's PY 2013 DSM programs.
107
Table 81. Nonresidential NTG
Table 82. Nonresidential NTG and Net Savings
S.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
Khawaja, The Cadmus Group, lnc
Schedule 1, Page 113 of 130
Table 83.2013 ldaho Net Savings
Residential
Nonresidential 15,595,342
4sero1
zJ"asgpas
__r_rlll
1oo%
es% ,
5.4 IRP Goals Achievement
Table 84 shows net evaluated savingt compared to the IRP goal of 19,009,200 kwh. The lRp goals are
set at the potfolio-level, ln order to conduct sector-level comparisons, Cadmus adopted the Avista
Business Plan goals by sector and applied those proportions to the IRP targets. PY 2013 achieved 115.7%
of the IRP target in ldaho with 2L,999,O99 kWh. Excluding the Residential Eehavior Program savings,
ldaho still met the IRP goal, at 100.5% with 19,128,194 kwh. Table 85 shows Avista's internal Business
Plan goal achievements,
Residential
Nonresidential 10,u9,696 13,436,118
Low lncome 462,495 499,901
123.8%
108.1%
:rr!i'-- 1oo=%-
708
Table 84. PY 2013 IRP Goal and Net Achieved Savings
Table 85, PY 2013 Avista Business Plan Goal and Net Achieved Savings
Nonresidential I 12,g!,8,322 13,430,118
S.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
Khawaja, The Cadmus Group, lnc
Schedule 1, Page 114 ot 130
Appendix A: Residential Billing Analysis Model Specifications
Overview of the PRISM Approach
A site-level modeling approach was originally developed for the PRISM software.ae ln this model, the
NAC is estimated separately for each customer account, for both the pre- and post-installation periods.
The weather normalization for each account and period relies on a longitudinal regression analysis. The
difference between the pre- and post-program NAC represents the program-related change in
consumption plus exogenous changes in consumption. Without a nonparticipant group this exogenous
change is not eliminated, but it is expected to be small for consumption over the three-year evaluation
period, especially with respect to the larger change in consumption from conversion.
Model Specificotion
Cadmus fitted each account with specific degree-day regression models, separately for the pre- and
post-installation periods. We first normalized the monthly bills by the number of days in each billing
period to obtain the average daily consumption (ADC). Then we calculated the average temperature
during each utility billing period.
This degree-day regression for each account is modeled as:
ADCx- c1 + B,AVGHDDiI * y;AVGCDDI * e1s
p.
Average daily kWh or therm consumption for each customer'i' during
billing month t
Participant intercepU represents the average daily kWh or therm base
load or the energy use for non-space heating or cooling purposes
Participant slope; represents the change in energy use for a unit change
in the HDDS
Base 55 average daily HDDs for customer 'i' in period t'
Participant slope; represents the change in energy use for a unit change
in the CDDs
Base 55 average daily CDDS for customer 'i' in period 't'
AVGHDDn =
AVGCDDt =
Cadmus used the results from the above estimation to compute the NAC for electricity:
NACI- di*365+ BiNORMHDDi + ?INORMCDDt
4e Fels et al- 1995
709
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 1, Page 1 15 of 130
Where:
NACI = Normalized annual kWh or therm consumption for each customer 'i'
d1 = The participant intercept; estimated from the above model
B, = The participant heating slope; estimated from the above model
NORMHDDI= Annual normal-year HDDS (base 65) for customer'i' in period f
?i = The participant cooling slope; estimated from the above model
NORMCDDi = Annual normal-year CDDs (base 55) for customer'i' in period 't'
Overview of the Regression Approach
Cadmus specified a conditional savings regression model with paired pre- and post-participation
months. This is a pooled regression approach that combines all participants and time intervals for a
single measure group into a single regression analysis. The observations vary across both time and
individual accounts. This pooled approach is recommended for cases like this, where there is no
separate comparison group and where other energy-efficiency measures are installed in homes.
Model Specificotion
Cadmus estimated a separate regression model for each of the groups. The model determined the ADC
of electricity of home 'i' in month 't' as:
ADCrt - a1*tg* PlHDDit * p2CDD11 * p3HDD11 rOtherit + B4CDDit *jtherrt + psPOSTit
* p5POSTlr+ HDD11 * p7PoST1.* CDDit + psPOSTtt +Otherrti €it
Where:
Or Average daily base load energy use in home'i'that is not sensitive to
weather or time. This analysis controlled for non-weather-sensitive and
time-invariant energy use with home fixed effects
Average energy use in month ? reflecting unobservable factors specific
to the month. This analysis controlled for these effects with month-by-
year fixed effects
Average daily usage per HDD and CDD (kWh or therm/degree day) in
the pre-conversion period
Average daily HDDs (heating load) during the billing cycle
Average daily CDDs (cooling load) during the billing cycle
Coefficients for HDD and CDD (kWh or therm/degree day) interacted
with the installation of other measures
An indicator variable for whether the month is pre- or post-installation
of other measure. This variable equals 1 in the months following the
maximum install date for all other measures, and equals 0 for months
prior to the minimum install date
tt
Fr F,
HDD
CDD
F, Fn
Other
770
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 1, Page 116 of 130
@
9t- It
POST
Coefficients used to estimate the conversion program effect on
electricity usage (as shown in next equation)
An indicator variable for whether the month is pre- or post-conversion.
This variable equals 1 in the months and years following the conversion
date, and 0 otherwise. The variable is defined using a combination of
Customer-Specific Measure lnstall Date and Full Year specifications
Error term for home 'i' in month 't'
Cadmus used the mean differences approach to estimate the above model. This approach removes the
customer-specific constant term, cr; and controls for the variation in electricity use between customers
and between months.
Cadmus estimated the fuel conversion program savings for each conversion group using estimated
coefficients on all the post-installation period dummy variable components in the above fixed-effects
regression model. For a home In conversion group 'j,'the gross savings are given by:
Savings;= pr*365+ p5AnnualHDD; * pTAnnualHDD; +ps*365
Where:
AnnualHDDj
AnnualCDDl
= Average annual HDDs for all customers in conversion group l'
= Average annual CDDs for all customers in conversion group l'
771
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule '1, Page 117 ol 130
Appendix B: Residentia! Behavior Program Data Cleaning Procedures
Cadmus conducted the following steps to inspect and clean the data provided by Opower:
1. Removal ofone customerfrom the opowerdata that appeared in both the control and
treatment groups.
2. Verification that customer assi8nments to treatment and control groups in the Opower data
corresponded to the assignments that Cadmus made. We found no discrepancies.
3, Removal of customers Opower flagged for exclusion from analysis because it was not possible to
generate an energy report or they received a report but were not randomly assigned.e
4. Checks for duplicate records. We found none.
One participant originally selected by Cadmus for the control group was missing from Opowe/s list of
participants, The Opower data also included 12 extra participants in the treatment group that were not
present in Cadmus'original sample, but Opower had flagged all ofthese to be excluded from the
analysis, After cleaning the data, there were 99,495 customers on Opower's list.
Cadmus conducted the following steps to clean the billing data provided by Avista:
1. Verification that customer account numbers were unique to addresses,
2. Removal of billing data for customers not in the Opower control or treatment groups and for
billing records ending before June 1, 2012 or beginning after December 31, 2013.
3. Removal of gas bills.
4. Removal of customers whose maximum daily average consumption in any billing period was
greater than 1,000 kwh per day. There were less than 10 such customers, and Cadmus assumed
their large bills were likely due to meter misreads, billinS errors, or significant commercial,
industrial, or agricultural activity which would make them ineligible for analysis. cadmus also
noted that there were 185 customers who regularly consumed more than 240 kWh per day on
average, but Cadmus did not remove these customers from the analysis.
5. Removalofduplicatebills.OneoftheadditionalbillingdatafilesthatAvistaprovidedincluded
many duplicate records; Cadmus did not include these in the analysis.
6. Removal of 50.00 bills. Cadmus noticed that there were many duplicate bills of this type.
Cadmus only removed these bills when either:
a. The service amount was 50.00 and the usage quantity (kwh) was non-zero, or
b. Both the service amount and the usage quantity were zero, but there was another non-zero
bill in the same period.
For example, some Avista staff requested to receive energy reports from Opower. There were 12 customers
who received reports but were not assitned to the treatment troup.
772
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 1, Page 118 of 130
Removal of August 2012 bills that ended on August 27, only when there were multiple bills for
that month. Many customers had two partially-overlapping bills in August 2012 that had the
same start dates. The first always ended on August 15 or 15, and the second always ended on
August 27. Cadmus noted that the next bill started on the 15 or 16 of August, not the 27, so we
removed the longer, partially-overlapping bill to ensure we would not be double-counting
energy usage.
Manual data cleaning of partially-overlapping bills. ln less than 20 instances, Cadmus manually
removed problematic partially-overlapping bills, so that we would not be double-counting
energy usage when summarizing the bills for analysis.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 1, Page 119 of 130
Appendix C: Residential Behavior Program Regression Mode! Estimates
Table 85 shows results from different panel regressions of home average daily electricity use. Cadmus
used Model 4 to estimate savings as shown in the report. There were only small differences between
models 1-4 in the estimated savings.
(o.oe)
o.741
io.rU(0.0s)
-0.5586 4.7672 4.7642 4.7637
Yes
l
Yes
Yes ]
54,324 l
Yes Yes Yes
No Yes No
No No Vei
54,g24 54,324 54,324
Notes: The dependent variable is the home's average daily electricity use for a month. cadmus based these
estimates on a Fin-D ordinary least squares regression of average daily consumption between June 2012 and
December 2013. The Huber-White estimated standard errors shown in parentheses are clustered on homes.
Table 86. Regression Estimates of Home Energy Report Effects on Energy Use
774
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
Khawaja, The Cadmus Group, lnc
Schedule 1, Page 120 of 130
S.
Appendix D: Low-lncome Weatherization Participant Survey
ln May 2013, Cadmus coordinated a phone survey of 150 residential low-income weatherization
program participants. We developed the participant survey instrument and defined the sample, then
subcontracted survey administration to an implementation firm.
Table 87 provides details regarding the planned and achieved completes for the telephone survey.
Total participants
screeneo out oue io a change in occupancy or incorreciphone numbei
Eligible participants on call list
Completed surueys
Sample size goal
Cadmus selected a random sample of participants from the PY 2012 Q3 to PY 2013 Q1 participant
population as available in April 2013 (434 participants). Cadmus aimed for and achieved 150 completed
survey responses, which provided results with 90% confidence and!5.L% precision atthe program level.
The survey achieved a high fielding response rate, as we used only 75% the sample frame to accomplish
the targeted completes.
We asked participants about their experiences with the program, addressing the following topics:
. Their previous awareness of the program, and how long they waited for an appointment and
pick-up.
. Functionality of equipment prior to repair or replacement
. Education they received through the program.
r Demographics and home characteristics
Progrom Aworeness ond WaitTime
Most survey respondents said they heard about the program through family or friends. Figure 20
presents all ways survey respondents heard about the program.
775
434
78
356
150
150
Table 87. Participant Telephone Survey Sampling Plan
S
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
Khawaja, The Cadmus Group, lnc
Schedule 1, Page 121 of 130
Figure 20. How Respondents Heard About the Program (n=125)
Fa milyf riends/word-of-mouth
Through another energy assistance
program or public serve aSency
ASency staff or Avista representathre
ItlenrpaperfiV/radio
Events
lnformation with my electric or natural 8as bill
Avista website
W 5%tfi 15%2(X 25%3ffi 3596 4U.45,6
Figure 21 shows how long respondents were on the waiting list for the program,
Figure 21. How Long Respondents Were on the Program Waiting list (n=142)
30,6
259(
2M
15%
109(
5%
Wo
OvBrturoyeaB Bctween one About sL months Lessthan sir
and two years to a year months
Nearly half of the respondents said they were on the program waiting list one year or less, with 25%
indicating they were on the wait list for less than six months. Thirty percent of the respondents waited
between one and two years, and 22% waited over two years for program services.
Previous ond New Equipment
Table 88 shows the distribution of installed equipment and the condition of the replaced equipment. For
respondents who received programmable thermostats, the table also indicates whether the installer
programmed the thermostat, educated the participant about how to install it, or neither.
776
Exhibit 3
Case Nos. AVU-E-14 AVU-G-'|4
S. Khawaja, The Cadmus Group, lnc
Schedule 1, Page 122of 130
E@
Table 88. Equipment lnstallation Rates and Equipment Condition
54%
24%
50%
29%
8%
Thermostat (n=143)s0%87%6%
For respondents who said their previous equipment had problems or did not work, Table 89 shows how
long the equipment was experiencing those issues.
Refrigerator 1n=150)
Furnace (n=146)
Water Heater (n=148)
Windows (n=148)
Doors (n=149)
Refrigerator (n=10)
Furnace (n=59)
Water Heater (n=34)
Furnace (n=51)
Water Heater (n=67)
L@6 6Wr
24% 6t%
32% 4t%
75%
50%
5t%
45%
62%
38%
61%
43%
7t%
92%
8%
L5%
7%nA
N/A
70/o
30%
t5%
26%
Table 90 details the fuel type ofold and replaced furnaces and water heaters for respondents who
received this new equipment to replace old equipment. The table does not include customers who did
not previously own a furnace or water heater before participating in the program
Electric
bas
oil
Electric
Gas
42%
53%
5%
76%
24%
to%
ga%
o%
25%
7S%
Progrom Educotion
Only 3% of respondents said they received little program information, while overtwo-thirds said they
received a lot of information, as shown in Figure 22.
777
Table 89. Equipment Problem Duration
Table 90, Furnace and Water Heater Fuel
S.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-'|4
Khawaja, The Cadmus Group, lnc
Schedule 1, Page 123 of 130
Figure 22. Amount of Much lnformation Respondents Received (n=119)
7ffi
ffi6
5096
4M
3096
2@5
l0/o
go
Alotofinformation? onlysorEinfonnation? orverylittle
information?
As shown in Table 91, 89% of respondents said they received educational pamphlets, and 97% of those
respondents said they read them.
Table 91. How Many Respondents Received and Read Pamphlets
89%Yes 97%
No !L%, 3%
Home Charocteristics
Figure 23 shows the distribution of when respondents' homes were built.
Figure 23. Year Respondents' Homes Were Built (n=141)
50,6
45%
M
35'6
36
25%
2M
1596
16
5%
M
Sefore Ectween Eetwn Eetween Betwe€n B€tween Eetwen19m lgDand 1951and 1970and lgS0and 199oand 2moand1960 1969 t9t9 1949 19!19 2m5
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 1, Page 124 ot 130
Most respondents live in a single-family home, mobile home, or trailer, as shown in Figure 24.
Figure 24. Home Types (n=147)
Sin8le-fdmily home Mobile home or Townhouse or Apartmenttrailer duplex(4orless buildingwithSor
total units) more units
Figure 25 shows that most respondents heat their home with natural gas, followed by electricity.
Figure 26 presents the distribution of respondents' primary heating equipment. Most respondents (59%)
said their primary heater is a natural gas furnace, followed by an electric furnace (22%1.
Figure 25. Heating Fuel (n=147)
AVo
7ffi
ffi
s$n
4Vo
3ffi
20/6
La6
M
tlatural Bas
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 1, Page 125 of 130
Figure 25. Primary Heater Type (n=147)
8ffi
7Cfr
ffi6
5096
4M
3ffi
2M
LW
MN
Gas Furnace Electric Wood Electric Heating Ol
Fumace Stove/Oven Baseboards EurrEr
Most respondents said that after the program equipment was installed, they either did not change or
turned down the temperature setting on their thermostat, as shown in Figure 27.
Figwe 27. Post-lnstallation Thermostat Changes (n=135)
lncreased the Decreased the Left the thermostat attemperature temperature the same setting as
before
Figure 28 shows what respondents use as a supplemental heating source. Most indicated using an
electric room heater or a wood burning device.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-I4
S. Khawaja, The Cadmus Group, lnc
Schedule 1, Page 126 of 130
Figure 28. Supplemental Heater Types (n=58)
Electricroom woodsto\re/ Gasfireplace/ PelletHeater Electricheater Oren/Firedace furn&e Fkeplace
Respondents who use a supplemental heating source said they used it less or about the same after the
program equipment was installed, as shown in Figure 29.
Figure 29. Post{nstallation Supplemental Heater Use (n=56)
@6
s096
40,6
30,6
2M
IM
M
About the sm amount
Figure 30 presents the distribution of equipment used to cool respondent's homes. When we asked if
they would change the way they cool their home after participating in the program, only 8% responded
affirmatively.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 1, Page '127 ot 130
Figure 30. Summer Cooling Equipment Types (n=140)
Windov room Fans or ceiling Central air- Air-sdrce heat Swamp coohrairconditioners f"ns conditioner pump (or evapoEtiw
coler)
Figure 31 shows the type of supplemental equipment respondents use to cool their home.
Figure 31. Supplemental Cooling Equipment Types (n=64)
ta6
7ffi
afr
s(fr
4M
30fr
20,6
IM
a6
Fanr or ceiling fanswindo{ rmm air- Central air- Air-source heatconditioneE conditioner pump
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 1, Page 128 of 130
E@
Appendix E: Low-lncome Weatherization - Billing Analysis Model
Specification
For each participant home, Cadmus estimated three models in both the pre- and post-periods in order
to weather-normalize raw billing data:
o Heating and cooling,
o Heating only, and
o Cooling only.
The heating and cooling PRISM model specification was:
ADCi, --a.+ BTAvGHDDit + PZATGCDDil +ei,
Where for each customer'i'and calendar month 't':
= The average daily kWh consumption in the pre- or post-program period
= The participant intercept; represents the average daily kWh base load
= The model space heating slope (used in the heating only and heating +
cooling models)
A\/GHDDi, = The base 55 average daily HDDS for the specific location (used in the
heating only and heating + cooling models)
f, : The model space cooling slope (used in the cooling only and heating +
cooling models)
AVGCDDil = The base 65 average daily CDDs for the specific location (used in the
cooling only and heating + cooling models)
€h = The error term
From the model above, we computed the NAC as follows:
NAC, -- a.*365+ \LRHDDi+ PTLRCDD,+ e,
Where, for each customer'i':
= Normalized annual kWh consumption
= The intercept that is the average daily or base load for each
participant, representing the average daily base load from the model
= Annual base load kWh usage (non-weathersensitive)
= The heating slope; in effect usage per heating degree from the model
: The annual, long-term HDDS of a TMY3 in the 1991-2005 series from
NOM, based on home location
ADCiI
cli
ft
NACi
d1
ai * 365
9,
LRHDDi
123
s.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
Khawaja, The Cadmus Group, lnc
Schedule 1, Page 129 of 130
Pt ,LRHDDI = Weather-normalized annual weather sensitive (heating) usage, also
known as HEATNAC
9, = The cooling slope; in effect, the usage per cooling degree from the
model
LRCDDT = The annual, long-term cDDs of a TMY3 in the 1991-2005 series from
NOM, based on home location
P2.LRCDDI = Theweather-normalizedannualweathersensitive(cooling)usage,
also known as COOLNAC
The error term
Although we used the same specification for both electric (non-conversion) and conversion participants,
Cadmus estimated separate fixed-effects CSA models for each group to determine program-level
savings:
AD C is = al * p $V G H D D E + PIAV G C D D i6 t psP O ST is + P4...AM t + €it
Where, for customer'i'and monthly billing period 't':
= Average daily kwh consumption during the pre- and post-program
periods
= The average daily kwh base load intercept for each participant (part of
the f ixed-effects specifi cation)
= The model space heating slope
= The average daily base-65 HDDs, based on home location
= The model space cooling slope
= The average daily base-65 CDDs, based on home location
= The kWh change in usage per day
= An indicator variable that is 1 in the post-period (after measure
installations) and 0 in the pre-period
= An array of bill month dummy variables (Feb, Mar, ..., Dec), 0
otherwisesl
6ft = The modeling estimation error
Cadmus estimated the above model for ldaho non-conversion and conversion participants separately,
The model coefficient, 6* is an estimate of the kWh savings per day in each model.
sl We excluded the January dummy variable from the independent variables, otherwise the 12 monthly
indiGtors would form perfect co-linearity with the intercepts; thus, the intercepts include the seasonality
from January.
ADCr
At
6t
AVGHDDfr
62
AV6CDDn
6z
PO'Til
MT
724
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 1, Page '130 of 130
MEMORANDUM
To: David Thompson, Avista
From: Danielle Kolp, Cadmus
Subject: 2013 ldaho Natural Gas Savings
Date: JuneL4,2OL4
This memorandum is intended to document the natural gas savings achieved by Avista Utilities' DSM
programs in ldaho for program year 2013. Though formal programs were suspended in ldaho for 2013,
there were several instances where gas savings were still achieved due to grandfathered projects or duel
fuel saving measures. The analysis methodologies for these savings are omitted from this memorandum,
but can be found in great detail in the Avista 20L3 Washington Gas Portfolio lmpact Evaluation report
submitted to Avista on May t5,201,4.
Total 2013 ldaho Natura! Gas Savings
ln 2013, Avista's ldaho service territory exhibited natural gas savings of 51.,772 therms across
nonresidential projects, residential measures, and the residential behavior program.
L8,L92
t,743
29,498
49,433
An Employee-Owned Company
www.cadmusgroup.com
18,580
2,56L
30,631
st,772
Nonresidential Savings
There were twelve natural gas projects in ldaho that were originated prior to 2013 but were physically
completed and paid incentives in 2013. These projects were subjected to the site visit and metering
sampling methodology along with the rest of Avista's natural gas projects included in the evaluation.
720 5W Washington Street
Suite 4oo
Portland, OR 97205
Voice: 503.467.7 I 00
Fax: [phone]
Corporate Headquarters:
1 00 5th Avenue, Suite I 00
Waltham, lvlA 02451
Voice: 61 7.673.7000
Fax:[phone]OO1
Exhibit 3
Case Nos. AVU-E-14 AVU-G-I4
S. Khawaja, The Cadmus Group, lnc
Schedule 2, Page 1 of3
Table 1. 2013 Reported and Gross Evaluated Savings for ldaho
Table 2. PY 2013 Nonresidential Gross Gas Savings
Table 2 shows the reported and gross evaluated savings for the 12 nonresidential projects in 2013,
resulting in evaluated savings of 18,580 therms yielding a t02% realization rate.
Prescriptive
Site Specific - HVAC
Site Specific - Other
Site Specific - Shell
Total
3
6
1
2
t2
Residential Savings
Though the residential natural gas DSM programs were suspended for 2013, 214 measures were
processed at the beginning ofthe year. The 99 clothes washer measures were actually processed as
electric measures, but upon evaluation, found to have natural gas water heating, so there were
additional gas savings from the electric dryer savings. Table 3 gives details on these six measures and the
reported and evaluated gross savings, which achieved 2,561 therms.
2,M7
L2,64L
27
3,O77
t8,t92
2,L35
13,3ss
26
3,064
1&s80
87%
to6o/o
96%
L00o/o
LO2%
Attic lnsulation with Gas Heat
Wall lnsulation with Gas Heat
Natural Gas Boiler
Natural Gas Furnace
Clothes Washer With Natural Gas
Water Heater
Simple Steps - Showerheads
Total
4
2
1
7
279
370
t4t
722
0
23L
L,743
279
370
t4l
722
420
630
2,56L
LOO%
IOOYo
too%
LOO%
N/A
272%
147%
101
2L4
720 SW Washington Street
Suite 400
Portland, OR 97205
Voice: 503.467.7'100
Fax: [phone]
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 2, Page 2 oi 3
Table 3. PY 2013 Residential Gross Gas Savings
Residential Behavior Savings
Avista began a residential behavior program in the summer of 2013 in both ldaho and Washington that
targeted electric savings, but Cadmus also evaluated gas savings achieved by the program. Cadmus
performed a billing analysis on the entire population of participating homes, and the evaluated savings
and confidence intervals can be seen in Table 4 below. ldaho homes participating in the program
reduced their gas usage by l.O4%.The gross reported savings are presumed to reflect the Avista
Business Plan assumption of L.0O% savings. The program achieved 30,631 therms in the second half of
2013 in ldaho.
Table 4. PY 2013 Residential Behavior Gross Gas Savings and Confidence lntervals
720 SW Washington Street
Suite 400
Portland, OR 97205
Voice: 503.467.71 00
Fax: [phone]
Exhibit 3
Case Nos. AVU-E-14 AVU-G-'|4
S. Khawaja, The Cadmus Group, lnc
Schedule 2, Page 3 of 3
ME
AVISTA2OL2.2OL3
PROCESS EVALUATION REPORT
May t5,2Ot4
Avista Corporation
1411 E Mission Ave
Spokane, WA 99220
The Cadmus Group, lnc.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 'l ot 127
An Employee-Owned Company . www.cadmusgroup.com
This page left blank.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-'|4
S. Khawaja, The Cadmus Group, lnc
Schedule3, Page2otl27
Prepared by:
Danielle C6t6-Schiff Kolp, MESM
Kate Bushman
Cameron Ramey
Allison Asplin
Hanna Lee
Andrew Carollo
M. Sami Khawaja, Ph.D.
Cadmus
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 3ol 127
This page left blank.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 4 ot '127
Table of Contents
Portfolio Executive 5ummary....... ........................ ix
Evaluation Activities ................ ix
Key Residential Findings..... ........................... ix
Residential Conclusions and Recommendations........ .......... x
Program Participation. ............................ xi
Program Design........... ............................ xi
Program lmplementation ................. ........................... xii
Marketing and Outreach ........................ xii
Key Nonresidential Findings.................... ........................... xii
Nonresidential Conclusions and Recommendations........ ........................xiv
Program Management and lmplementation............ .........................xiv
Customer Feedback ......... xv
Market Feedback ............. xv
Marketing and Outreach ........................xvi
Quality Assurance and Verification.................... ..........xvi
Residential Process Report ........... ........................ 1
Program Overview .............2
Evaluation Methodology and lnformation Sources.... .........................5
Status of Evaluation Recommendations.............. ........ 13
Program Participation ............L4
Savings and 1ncentives.................... .......L4
Participation Trends........... ...........,........15
Program Design, Management, and lmplementation.......... ............-.......22
Effectiveness of lmplementers................. .......................... 28
Opower ............................28
Residential Behavior Program Description... ...............28
Residential Behavior Program lmplementation ................. ...............29
Future of the Residential Behavior Program ...............30
Data Tracking Summary...... ...................31
Planned Changes in Avista Data Tracking ....................33
Marketing and Outreach ........33
Marketing Approach ........33
Marketing Objectives and Strategies.................... .......34
Planning and Processes ..........................34
Target Audience and Customer Motivators ................35
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 5 ot'127
Outreach Channels....... .......................... 35
Every Little Eitand Efficiency Motters Campaigns.... .........................35
Materials and Messagin9.................... ......................... 37
Marketing Execution and Measurement................ ...........................37
Sources of Participant Awareness .........37
Avista Customer Awareness of Energy-Efficiency Rebates..................... .................39
Participant Experience and Satisfaction.................. ...........40
Overall Program Satisfaction ..........,......41
Rebate Amount and Promptness Satisfaction .............42
Residential Program Freeridership and Spillover. ..............45
Freeridership ....................45
Spillover ...........................48
Residential Conclusions and Recommendations........ ........49
Program Participation . .................,.........49
Program Design........... ...........................50
Program lmplementation ................. ...........................51
Marketing and Outreach ........................51
Nonresidential Process Report ...........................52
Program Overview ...........52
Evaluation Methodology and lnformation Sources.... .......................54
Status of Evaluation Recommendations.............. ........ 59
Program Participation ............59
Savings and 1ncentives.................... .......59
Program Design, Management, and lmplementation.......... ....................50
Overview...... ....................50
Program Logic ............. ...........................50
lnternal Communication................. .......62
Effectiveness of lmplementers................. .......................... 53
Data Tracking, Verification, and Quality Assurance ..... ............................63
Participant Characteristics, Experience and Satisfaction ................. ........54
Participant Characteristics .................... ....................... 55
Participant Satisfaction ... ....................... 55
Program Barriers......... ...........................68
Program Benefits .............69
Market Feedback...... ..............70
Contractor Awareness ........................... 70
Program lmpact on Sales .......................71
vt
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 6 ol 127
Market Transformation.............,...... ............................72
Marketing and Outreach ........73
Program Marketing Approach ...............73
Customer Awareness .... ......................... 75
Nonresidential Program Freeridership and Spillover. ........77
Freeridership ....................77
Spillover ...........................80
Nonresidential Conclusions and Recommendations........ ........................ 80
Program Management and lmplementation............ .........................80
Customer Feedback .........81
Market Feedback .............81
Marketing and Outreach ........................81
Quality Assurance and Verification... ...........................82
Appendix A: Status of PY2010 and PY2011 Residential Evaluation Recommendations.............................83
Appendix B: Status of PY2010 and PY2011 Nonresidential Evaluation Recommendations.......................86
Appendix C:2072 Nonresidential Process Evaluation Memorandum ............90
lnterview Findings: Large Project Review Challenges and Changes. .......................90
Review Process Challenges ldentified by Avista ..........91
lssues ldentified Through the Large Project Review........... ...............92
Planned Process lmprovements .................... ..............92
2OLL-2OL2 Database and Realization Rate Review ............93
Database Review........... .........................94
Realization Rate Review.. .......................98
Conclusions and Recommendations .......... 105
Large Project Review Process ..............106
Database and Realization Rate Review ......................107
Memo Appendix A.............. ........................109
vil
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 7 ot 127
This page left blank.
vlil
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 8 ot'127
Portfolio Executive Summary
Avista Corporation contracted with The Cadmus Group, lnc., to perform a portfolio-wide evaluation for
the 2012-2013 demand-side management programs. This report presents the process evaluation
findings for the residential and nonresidential sectors.
Evaluotion Activities
Table ES-1 summarizes the process evaluation activities conducted by sector.
Avisla Program Staff lnterviews' 7
Third-Party lmplementer lnterviews L
,t
1,00s
2,t50
20
270
Nonparticipant Surveys .
A:se!! Lellgl rrallr n c !!q!1ry s
Review of Program Documentation
140
Review of Marketing Materials
Review of Stakeholder Reports
Multiple representatives present for some interviews.
Key Residential Findings
The residential process evaluation resulted in the following key findings for the programs examined
(listed in Table ES-2).
Table E5-2. PY?OL2 - PY2013 Residential Programs
ENERGY STAR Products
H igh-Efficiency Equipment
Home Audit
Manufactured Home Duct Sealing
Residential Behavior
Weatherization and Shell
Electric-Only Programs
Second Refrigerator and Freezer Recycling
Simple Steps, Smart Savings
Space and Water Conversions
Table ES-1. PY 2OL2-207? Process Evaluation Activities
lx
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 9 of '127
Participation levels in many of Avista's residential programs trended downward during PY2012
and PY2013. Many factors contributed to the downward trend, including reduced measure
offerings and the 2013 discontinuation of natural gas incentives in ldaho. The trend
experienced by Avista's programs is similar to participation trends in other regional utility DSM
programs.
The Simple Steps, Smart Savings program saw increased participation, partly due to new
measure offerings. Energy-efficient showerheads were added in 2OL2 and LEDs were added in
2013.
Avista's overall program design is effective, but there is room for improvement around internal
communication between Avista staff.
Avista staff showed a strong commitment to customer satisfaction, achieving fast rebate
processing despite increasing complexity of applications. Avista staff have also taken steps to
improve data tracking, such as integrating additional program data into a central database.
ln addition, program marketing through mass media channels had to be tailored to avoid
customer confusion about different incentive offerings in ldaho and Washington.
Key sources of program information for customers included contractors (L7% in 2OL2; 28% in
2013), bill inserts (15%; LGTol, and word of mouth (10%; 14%). Changes in information sources
reflected changing program offerings such as the elimination of appliance rebates in 2013.
General population awareness of Avista's rebates decreased from53%in20t2lo54% in 2013.
Bill inserts are the most common way for the general population to learn about Avista's
rebates.
o Participant satisfaction increased since the 2011 process evaluation, with 89% of 2013
participants being "very satisfied" with their program experience. Only a small number of
customers expressed any level of dissatisfaction across the three years in which Cadmus
conducted surveys.
o Avista's appliance rebates experienced a high level of freeridership, likely due to high market
penetration of ENERGY STAR appliances and comparatively low incentive amounts-as a
percent of incremental cost. Avista adjusted their program offerings to reflect this market,
discontinuing appliance rebates in 2013.
. Many of Avista's customers - both participants and nonparticipants - reported installing
additional energy-saving improvements without receiving any rebate because of Avista's
programs' influence. These actions contribute to program spillover. Out of the 3,215
customers Cadmus surveyed in 2OL2 and 2013, 113 (or roughly one in every 28 customers)
reported a spillover measure.
Residentiol Conclusions qnd Recommendqtions
This section describes the evaluation's conclusions and recommendations for the residential programs.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-l4
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 10 ot'127
Program Participation
Conclusion: Avista's implementation of new and continued support for existing third-party implemented
programs such as Simple Steps, Smart Savings and Residential Behavior effectively captures energy
savings in the residential market segments.
o Recommendotion: Continue exploring new measures, program designs, and delivery
mechanisms that leverage the national expertise of experienced third-party implementation
firms. Possible programs may include additional partnership with ENERGY STAR in the form of
the Home Performance with ENERGY STAR program.
Conclusion: Avista's continued investment in pilot programs provides a low-risk way test the
effectiveness of new measure offerings, delivery channels, and implementation partners.
o Recommendqtion: Continue testing new program designs and measure offerings through the
use of pilots-even if secondary sources of funding or local partners are not available.
Conclusion: While still early, evaluation findings indicate the Residential Behavior program is an effective
way to capture savings in the residential market and Opower is a strong partner for program
implementation.
o Recommendation: lf determined to be cost-effective, consider expanding the Residential
Behavior program (for example, lowering the energy consumption threshold for participation)
and implementing measures to track the methods these customers use to save energy. Given
that Avista has already included all cost-effective customers in their target population for this
program, future opportunities for expansion may be limited.
Program Design
Conclusion: lnconsistencies continue to exist in measure and program naming and organization across
program planning, tracking and reporting activities which result in less transparency in program
operations and limit effective program evaluation.
. Recommendotion: As part of the transition to the new data tracking system, consider aligning
program and measure names with offerings articulated in annual business plans and other
planning materials.
Conclusion: Reduction in Avista natural gas rebates and elimination of appliance rebates give customers
fewer ways to participate in Avista energy-efficiency rebate programs.
. Recommendotion: Consider ways to encourage repeat participation (such as marketing targeted
at previous participants and online profiles that reduce application paperwork).
Conclusion: Considering self-report customer freeridership scores and market baseline data from the
RTF is an effective way to assess the appropriateness of measure offerings.
xl
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 11 ot 127
. Recommendation: Continue use of customer freeridership and market assessments as a way to
assess the appropriateness of measure offerings.
Conclusion: Many ongoing changes in Avista's program design and measure offerings are driven by the
need to continue to meet cost-effectiveness requirements. Avista's examination of measure and
program-level cost-effectiveness will determine the character of its portfolio in future program years.
o Recommendotion: Develop a transparent process for assessing measure or program cost-
effectiveness and communicating results internally. Consider ways to ensure high-quality cost-
effectiveness analysis that aligns with industry best practices, such as obtaining an objective
third-pa fi review of current cost-effectiveness screening processes.
Program !mplementation
Conclusion: Avista prioritization of customer satisfaction has been very successful and overall participant
experience is very positive across all rebate programs.
. Recommendotion: Continue Avista's commitment to customer satisfaction, but monitor:
lncreased staffing costs; and
lmpacts of the 90-day participation window on freeridership.
Marketing and Outreach
Conclusion: Avista implements a strong general awareness campaign around energy-efficiency, but
some room exists in market segmentation and targeting specific customer groups.
. Recommendation: Utilize survey results from this evaluation and other data collection activities
to understand which audiences are more likely to participate in Avista programs.
Key Nonresidentiol Findings
The nonresidential process evaluation resulted in the following key findings for the programs examined
(listed in Table ES-3).
Table ES-3. PY2OL2 - PY2013 Nonresidential Programs
Prescriptive Program
xt,
Exhibit 3
Case Nos. AVU-E-14 AVU-G-'|4
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 12ot 127
Night Covers for Refrigerated Cases
caie lichtins
Strip Curtains for RetrgeraGa Spices
lnsulation for Suction Lines
ioi water ranti
Program participants were more likely than nonparticipants to own their facilities: according to
surveys 178% of participants owned their facilities, compared with 67% of nonparticipants).
Overall, participants reported high satisfaction ratings. The vast majority were "very satisfied":
87o/o for Prescriptive, 75% for Site-Specific, and 88% for EnergySmart Grocer. Only a handful of
customers (roughly 1%) reported any level of dissatisfaction.
All three nonresidential programs received the same satisfaction ratings or better than they did
in 2OLL, with the EnergySmart Grocer program showing a 23% increase in "very satisfied"
customers over 2011.
Though still showing high overall satisfaction, the Washington Site-Specific program had the
lowest level of "very satisfied" participants at690/o. Among these participants, lower levels of
satisfaction stemmed from inadequate information included in the program materials, and a
lower-than-desired rebate amount. However, satisfaction with Avista's staff remained high
despite these minor issues: 90% or more of participants in every category were "very satisfied"
with staff.
Contractors were the primary source of program information for nonresidential program
participants (37%. Other common sources of information were word of mouth (23%) and direct
contact with Avista (17%).
Among nonparticipants, awareness of Avista's energy-efficiency rebates has remained fairly
constant since 2010, with around 4 in 10 nonparticipants being aware of the programs (38% in
2013).
Avista's management and implementation of DSM programs has had some persistent
organizational challenges, which may have impacted the effectiveness of implementation
processes. While not limited to any specific part of Avista's DSM staff, many of the issues have
primarily affected the nonresidential program processes.
Cadmus' review of Avista's implementation and AA/aC processes showed that the accuracy of
project savings estimates has increased since 2011, there is still room for improvement. Figure
xiii
Exhibit 3
Case Nos. AVU-E-'|4 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 13ol 127
ES-1 shows the percentage of electric realization rates for site-specific projects that fell within
the range of 9O%lo 77O%. This range indicates a good level of accuracy in reported savings.
Figure ES-1. Nonresidential Site-Specific Project Electric Realization Rates 2011-2013
7Vo
6Mo
SVo
4Mo
3Mo
zffio
lWo
o%
120tL
a2012
2013
Realization Rate = 9G11096
Cadmus' interviews with lighting contractors - conducted as a supplement to the ongoing Panel
Study research - revealed that Avista's programs increase sales of energy-efficient lighting
equipment for both participating and nonparticipating contractors: 15 out of 20 reported that
their sales increased because of Avista's programs.
The prescriptive program showed 9% freeridership in 2013, showing a large decrease in
freeridership as compared to the 2011 result. The site-specific program showed 30%
freeridership in 2013, showing an increase as compared to 2011.
Nonresidential Conclusions ond Recommendotions
This section describes the evaluation's conclusions and recommendations for the nonresidential
programs.
Program Management and lmplementation
Conclusion: Several parties over several years, internal and external to Avista, have observed the need
for greater data quality assurance, in both documentation and input tracking. Quantitative inputs to the
savings and rebate calculations have repercussions for tariff compliance,l incentive payments, and
savings realization rates.
1As noted in ldaho Public Utilities Commission Order Number 33009 on Avista Corporation's Application for a
Findingthat it Prudently lncurred its 2010-2012 Electricand Natural Gas Energy Efficiency Expenditures.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 14 ol'127
xtv
. Recommendotion: Avista should continue efforts to improve program processes. Cadmus
understands that a reorganization of the DSM group has occurred concurrent to the delivery of
this report. This change may be an opportunity for fresh perspectives, clarified responsibilities,
and improved coordination within and between teams. We believe unifying the organizational
structure under central leadership is a step in the right direction and may help alleviate some
previously documented issues with internal communications.
In addition to the reorganization, Cadmus recommends that Avista develop standardized
processes within the DSM group, including clear delineation of roles and precise description and
assignment of all processes and responsibilities for both residential and nonresidential
programs. All affected parties should be included in formalizing and standardizing the DSM
group's processes, roles, and responsibilities. Further, all parties must formally agree to clearly
delineated responsibilities under the new organizational structure. While these activities need
to be prescriptive and precise, we caution that the resulting structure should still allow some
flexibility: increased clarity, transparency, and accountability should serve to enhance program
delivery and customer satisfaction.
Customer Feedback
Conclusion: Customers were highly satisfied with the program overall and with individual components.
Customer satisfaction has increased since 2011, which had in turn increased from 2010.
. Recommendation: Continue to prioritize and monitor program satisfaction.
Conclusion: Customers appeared to be slightly less satisfied with the Washington Site-Specific program
than with other programs. The largest source of lower satisfaction was the participants' reactions to
program materials. Many customers said they received no program materials, and many participants
learned about the program from their trade allies.
o Recommendotion: Consider taking action to strengthen the use of program materials. Consider
providing trade allies with printed program information flyers or brochures to give to customers.
Maintaining up-to-date information for trade allies is critical when they are the key party
delivering the program's message and participation details.
Market Feedback
Conclusion: According to commercial lighting contractor feedback, the nonresidential programs are
successful in driving incremental energy-efficient equipment sales, and the market has not yet
transformed to make energy efficiency standard practice.
o Recommendotion: Continue to monitor market transformation indicators to measure programs'
market impact over time.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 15 ot 127
Marketing and Outreach
Conclusion: The characteristics of Cadmus' survey respondents indicate that the office / professional
services and local government sectors may be underserved by the programs relative to their incidence in
the nonparticipant population. Further research is necessary to determine whether this is true.
c Recommendotion: ldentify underserved industries, and seek opportunities to target outreach to
specific underserved industries:
I nvestigate overa I I customer industry distribution
Compare to participant industry distribution
Develop targeted outreach strategies for any underserved sectors
Quality Assu rance and Verification
Conclusion: Avista monitored its site-specific project review process and instituted refinements during
the evaluation period in response to feedback from users. While this has led to improvements, including
notably improved reliability of reported savings in2OL2, quality assurance problems may persist.
o Recommendotion: Continue to monitor the effectiveness of the site-specific project review
process and refine as needed. Cadmus recommends implementing the following to ensure
continued improvement:
AII large prescriptive or site-specific projects reporting savings over a threshold of 300,000
kWh or 10,000 therms should undergo a complete AA/aC review prior to incentive payment
in addition to the standard Top Sheet review process. Typically, a QA/QC process reviews
engineering calculations, verifies inputs, checks payback period and incentive payments for
reasonableness, and ensures compliance with program requirements and tariff rules. ln
order to align with the above recommendation regarding program management and
implementation, Cadmus recommends that Avista determine and document the specific
requirements and steps in the QA/QC process through a collaborative process that will
ensure accountability and balance needs for efficienry and customer satisfaction.
Conduct an external third-party review of Top Sheets, including reviewing a random sample
of completed Top Sheets for completeness and accuracy. These were not reviewed as part
ofthe current process evaluation, but should be included in the next process evaluation.
Review should not only verifo the presence of the Top Sheets, but also the quality and
accuracy of the information provided.
xvt
Exhibit 3
Case Nos. AVU-E-'|4 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 16 ot'127
Residential Process Report
lntroduction
This residential process evaluation focuses on ten Avista programs offered to ldaho and Washington
natural gas and electric customers during program years 2OL2 and2OL3 (PY1OL}and PY2O13).2 ln this
evaluation, Cadmus sought to address the following researchable questions:
o What are the major trends in measure offerings and program uptake, and how do they compare
to other utilities?
o What barriers exist to increased customer participation, and how effectively do the programs
address those barriers?
o How satisfied were customers with the programs?
r What changes to design and delivery would improve program performance?
ln assessing these topics, Cadmus relied on three main data collection efforts:
o Review of program tracking data, documents, and invoice materials;
o lnterviews with Avista and third-pafi program implementation staff; and
o Telephone surveys with participating and general population3 customers.
ln this effort, Cadmus sought to align evaluation resources with evaluation objectives and focus on areas
of uncertainty and programs with higher reported gross savings. Therefore, as indicated in Table 1,
evaluation activities generally centered on programs implemented directly by Avista (rather than a
regional partner) and established programs rather than pilots. Table 3 provides additional detail on the
scope of evaluation activities applied to each program.
ENERGY STARO Homes Limited
ENERGY STAR Products
High-Efficiency Equipment
Home Audit Limited
tvl.r*f*t,r*d H"." D,*t S..lrg Limited
'Not all programs are offered to customers in both states. For example, the Home Audit program operated only in
Spokane Washington. Avista's programs operate on calendar years, with program years running from January
through December.
' ln 2}L2and 2013, Cadmus surveyed a random sample of Avista Washington and ldaho customers. Cadmus did
not implement any screens for program participation when sampling, so it follows that some percentage of
respondents have at one time participated in an Avista energy-efficiency program.
Full
Full
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 17 ol 127
Table 1. PY2OL? - PY2013 Process Evaluation Scope
Natural Gas and Electric Programs
I Residential Behavior
i Weatherization and Shell Full
j!:4r,.9{yrlogra[:__ __ ____l--:*qal"tne:;t"l:g Fr-r;;;; i"-.yd,al " gL__ --']
; Simple Steps, Smart Savings I Limited __ iI Sp.." and Water Conversions Full I
ln addition to the programs identified in Table 1, Avista offers energy-saving opportunities to residential
customers through CFL Geographic Saturation events and Aclarao Software Applications. As energy
savings from these activities are generally low (CFL Geographic Saturation events) or not tracked
(Aclara), Cadmus did not review them as part of this evaluation.
Program Overview
The following section briefly describes the programs reviewed in this evaluation.
ENERGY STAR@ Homes
The Northwest Energy Efficiency Alliance (NEEA) administers a regional ENERGY STAR Homes Program,
which Avista supports. When a home in Avista's territory makes it through the program and is certified
as ENERGY STAR-compliant, Avista pays a rebate to the homebuilder. The amount of the rebate is based
on Avista fuel-service(s) used in the home.
ENERGY STAR Products
This program offers direct financial incentives to motivate customers to purchase and install energy-
efficient appliances. The program indirectly encourages market transformation by increasing demand
for ENERGY STAR products-specifically, appliances such as refrigerators and clothes washers.
High-Effi cie ncy Equipment
This program offers four incentive categories for electric and gas customers seeking to purchase:
o High-efficiency water heaters;
o High-efficiency natural gas furnaces or natural gas boilers;
o High-efficiency air-source central heat pumps; and
. Primary heating systems incorporating a variable-speed motor.
Prior to 2011, these measures were offered under the Water Heating and Heating and Cooling Efficiency
Programs.
Home Audit
The Home Audit Program, launched in May 2010 and implemented with support from municipal
partners, sought to determine home energy audits' cost-effectiveness for capturing electric and gas
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 18 of 127
savings. Eligible Avista customers must have resided in single-family homes, duplexes, or manufactured
homes located in Spokane County. The program offered energy audits to customers, conducted by
Building Performance lnstitute (BPl)-certified auditors, at no cost to eligible customers. An Energy-
Efficiency Community Block Grant (EECBG), under the American Recovery and Reinvestment Act (ARRA),
partially funded this program. The program operated through PY?OL2.
Manufactured Home Duct Sealing
This program, launched in October 2012, provides duct testing, sealing, and repair to Washington
customers in electrically heated homes located in Adams, Asotin, Ferry, Franklin, Garfield, Lincoln,
Spokane, Stevens, and Whitman counties. This program is offered free of charge to customers, with 50%
of the funding coming from Avista's DSM funds and 40o/o provided through the Washington State
University (WSU) Community Energy Efficiency Program (CEEP). Allwork is performed by UCONS LLC
(UCONS), a third-party contractor.
Residential Behavior
The Residential Behavior Program is a peer-comparison program that began in spring 2013 and is
scheduled to continue through 2015. Through the program, residential customers receive regular
reports on their energy usage and comparisons to the usage of other customers in their immediate
vicinity. Avista expects the program to increase the participation in their residential rebate
programs and encourage behavior changes that result in kWh and therm savings. The program is
offered at no cost to a sample of customers preselected by Avista (with assistance from Cadmus
and Opower) and is implemented by Opower.
Weatherization ond Shell
This program offers incentives for attic, wall, and floor insulation measures, and is available to
residential electric and gas customers with homes heated with an Avista fuel.
Second Refrigerator and Freezer Recycling
This program, available to Washington and ldaho electric customers, provides financial incentives to
customers recycling refrigerators and freezers. The program seeks to reduce energy consumption by
recycling up to two inefficient secondary refrigerators or freezers per home. JACO Environmental, lnc.
(JACO), the implementation contractor, is responsible for scheduling, pick-up, recycling, rebate
payment, and data tracking.
Simple Steps, Smort Savings
Avista sponsors an upstream, buy-down program, administered by the Bonneville Power Authority (BPA)
and implemented by CLEAResult (formally Fluid Market Strategies). The program, available to customers
in Washington and ldaho, offers discounted twist and specialty CFLs, LEDs, and energy efficient
showerheads at many large retail locations.
Space ond Wdter Conversions
This program offers incentives for three types of conversion:
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 19 ot 127
r Replacement of electric resistance heating equipment as a primary heat source (either electric
forced-air furnaces or electric baseboard heat), with central, natural gas heating systems;
o Replacement of electric resistance heating equipment with central heat pumps; and
o Replacement of electric water heaters with new, natural gas water heaters.
Table 2 lists the residential energy-efficiency programs offered in PY2012 and PY2013-along with their
associated measures and incentives.
Table 2. PYzOLz - PY2013 Residential Programs and lncentives
, ENERGYSTARHoMES
ENERGY STAR Home with Electric-Only or Electric and Gas s900 s6s0
ENERGY STAR Home with Gas-Only Ssso
ENERGY STAR PrOduCtS
ENERGY STAR Freezer SZO N/A'lirrr-.rnrr{ :.- -rr:I=-_yienenev srnn oitr*r.rt'", -- - - Szsl -- ---- - itlt.
ENERGY STAR Clothes Washer
I High-Efficiency Equipment
1 High-Efficienry Natural Gas Boiler or Furnace
s6s0
Szs
High-Efficiency Air Source Heat Pump
Ductless Heat Pump
s400 I slOO
N/A
)
slOO
S2oo
s10oVariable Speed Motor
Sso
High-Efficiency Natural Gas Water Heater
Home EnergyAudit
Home Audit
i Manufactured Home Duct Sealing
Sso
No cost to customer
i DuctTesting, Sealing, and Repair No cost to customer
I Residentiat Behavior
Participating Customer No cost to customer
i Weatherization and Shell
Attic lnsulation
Wall lnsulation
50.25 per sq. ft.
$0.50 per sq. ft.
$0.50 per sq. ft.So.5o per sq. ft. l
Sloo
i Floor lnsulation
Fireplace Damper
Space and Water Conversions
__ __l
$0.25 per sq. ft.
1
-so.so
p"r t-c. fL I
i Electric to Natural Gas Furnace sTso s7s0
Electric to Air Source Heat Pump s7s0 s7s0
Electric to Natural Gas Water Heater s200 s200
Exhibit 3
Case Nos. AVU-E-14 AVU-G-l4
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 20 ot 127
iecona Refrigerator and Freezer Recycling
Appliance Recycled
Simple Steps, Smart Savings
Showerhead
Light-Emitting Diode (LED)
Compact Fluorescent Bulb (CFL)
Variable upstream buy-down
"N/A" indicates measure offering was eliminated. However, some rebates may have been paid in the
early months of the year, as Avista offers customers a 90-day grace period between project completion
and when rebate materials must be submitted.
Evaluation Methodology and lnformation Sources
Cadmus' approach to this residential portfolio-wide process evaluation relied on three main reviews and
data-collection efforts. Table 3 indicates which data-collection activities we applied to each program.
Manufactured Home Duct Sealing
Opower
Weatherization and Shell
Electric-Only Programs
Second Refrigerator and Freezer Recycling
Simple Steps, S..tt S*ings
Space and Water Conversions
*Customer surveys asking specifically about program participation. All residential customers groups
targeted in general population studies.
A description of each activity follows below.
Moteriols and Database Review
Cadmus' document review focused gaining an up-to-date understanding of PY2OL2 - PY2013 program
offerings, planning assumptions, participation, and marketing methods. Our review centered on the
following materials:
. Avista's in-house tracking database;
Sso Sso
//
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 21 ol 127
Table 3. Data Collection Activities Applied to Each Program
Natural Gas and Electric Programs
. UCONS' duct sealing tracking data;
o JACo's appliance recycling tracking database;
o CLEAResult invoice summaries;
o Avista's PY20L2 and PY2013 DSM Business Plans;
o An internal Avista program implementation manual;
o Avista marketing collateral;
o The Everylittlebit.com website; and
o TheAvistautilities.comwebsite.
Progrom Stafl ond Market Actor lnteruiews
lnterviews with program staff and market actors provided first-hand insights into program design and
delivery processes, and helped evaluation staff interpret the information collected. We conducted
program staff interviews in two rounds, one in January 2013 and another in January and February 2014.
Table 4 provides a summary of interview data collection.
Cadmus interviewed six members of Avista's Washington and ldaho program staff, including:
. Demand-side management (DSM) program managers;
o Planning, Policy, and Analysis (PPA) team members; and
o Marketing staff.
Cadmus conducted these interviews in person in2OL2 and by phone in 2013, using prepared interview
guides. When necessary, Cadmus requested clarifying information via phone or e-mail. Staff interviews
addressed the following topics:
o Changes in measure offerings;
o Goals;
. Program design;
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 22 ot '127
Table 4. PY2OL? - 2013 Program Staff lnterviews
Avista Policy, Planning, and Analysis Staff
Residential Behavior lmplementation (Opower) Staff
* Multiple non-Cadmus staff participated in interview.
. lmplementation:
r Marketing
' Target markets
. Tracking; and
. Qualiry assurance and control (aA/aC) procedures.
Cadmus conducted only one interview with staff representing third-party implementation companies.
We determined that this was appropriate for the following reasons:
o Cadmus interviewed representatives from Opower, the Residential Behavior Change program
implementer, as this is a new program with high levels of participation.
o Staff from JACO and CLEAResult participated in in-depth interviews in 2OL2 (to inform the
PY2OLL evaluation effort) and interviews with Avista staff identified few program changes and
limited issues.
o Cadmus did not interview staff implementing the Home Audit or the Manufactured Home Duct
Sealing program. The Home Audit program completed in PY2013, and the Manufactured Home
Duct Sealing Program is not expected to continue beyond PY2OL4.
The interview centered on the following topics:
r Goals;
. Program design;
o lmplementation;
o Marketing; and
. QA/QC.
Participating and Generol Populotion Customer Telephone Surveys
Telephone surveys constituted a large part of PY?OL2 - PY2013 evaluation data collection activities,
informing both impact and process evaluations of several programs. When conducting surveys, we took
special care to address potential issues of bias in the following areas:
o Sample selection (which customers to include in the survey sample frames);
o Responses (are customers answering the survey as a group representative of the sample frame);
and
o Data analysis and reporting (analysis conducted with an appreciation for the sample selection
and limitation of survey data collection).
We conducted all surveys with the assistance of several subcontracted market research firms, selected
for their experience with different data collection techniques and market segments.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 23ol 127
Participating Customer Surveys
Participant telephone surveys offered important insights into program experiences for six residential
measure categories (five programs),4 exploring the following topics:
o Source(s) of program awareness;
o Satisfaction;
o Awareness of energy efficiency;
o Participationbarriers;
o Freeridership and spillover; and
o Customercharacteristics.
Cadmus conducted the participating customer surveys in two rounds, one in March and April 2013 and a
second in February 2014. This approach ensured that respondents would have a clear recollection of
their participation experience. Table 5 provides a summary of unique customers (identified using Avista
account number) and surveys completed in each effort.
Table 5. Residential Participant Details and Survey Sample (lD and WA)
Natural Gas and Elecilic Programs
ENERGY STAR Products 5,429 L49 2%55 8%l
Heating and Cooling Efficiency 3,747 742 4%)2,490 70 3%,
Water Heating 629 88 14% |315 60 L9% I
Weatherization and Shell
Measures
Etectric-Only Protrams
313
- -,i
,_-__.1
702 \5%
I
, _ __-__i
*r
-'ii]
Second Refrigerator and
Freezer Recycling
Space and Water Conversions 34 20%24%
Total 7%l
Cadmus designed participant survey completion targets to yield results with 90% confidence and t10%
precision levels, for measure-category level survey results.ln2OL2, we expanded this approach to yield
results at the measure category and state level. Cadmus deemed this necessary as data collected
through these surveys-specifically installation rates-were used to inform an impact assessment of
o ln 20U, Avista combined the Heating and Cooling Efficiency and Water Heating Programs into a single program,
High Efficiency Equipment. Given the differences in these measure types and to ensure comparability to survey
data collected for earlier evaluations, survey targets and analysis for these respondents remain separated.
1,351 133 tO%
171
Exhibit 3
Case Nos. AVU-E-14 AVU-G-l4
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 24ol 127
65 5%
Avista's residential programs. The participant survey sampling plan also drew upon multiple factors,
including feasibility of reaching customers, program participant populations, and research topics of
interest.
Cadmus did not conduct participant surveys with Simple Steps, Smart Savings customers, as that
program has an upstream focus and therefore does not track participant contact information. Similarly,
for ENERGY STAR New Homes, Cadmus did not survey residential customers purchasing rebated homes
because the program paid rebates to builders, not to end-use customers. Cadmus also did not focus
evaluation resources on new programs that are subject to review by their own implementation
organizations (i.e., Residential Behavior) or temporary programs (e.9., Home Audit).
Within each program stratum, Cadmus randomly selected program participant contacts included in
survey sample frames. A review of collected data shows geographic distribution of survey respondents
clustered around urban centers, specifically the cities of Spokane, Coeur d'Alene, Pullman, Moscow, and
Lewiston. This aligns with population distributions in Avista's service territory. Figure 1 provides the
distribution of participating customer survey respondents.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 25ol 127
Given the wide range in program sizes, we weighted survey responses by participation (i.e., unique
customers in each measure category) when reporting responses in aggregate, thus ensuring feedback
represented the overall population. Table 5 shows the weighting scheme applied to PY2012 - PY2013
survey frequencies. Findings from PY2011 surveys included in comparisons also include post-survey
weightings.s
t Avisto 2077 Multi-Sector Process Evoluotion Report. Cadmu s.zoLz.
10
Exhibit 3
Case Nos. AVU-E-'|4 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 26 ot 127
Table 6. PY2OL? - 2013 Participant Survey Sample Design and Weights by Program
2012 Population and Achieved Surveys
ENERGY STAR PTOdUCtS 6,429 749 43.15
26.39and Cooling Efficiency 3,747 142
629 88 7.ts
and Shell Measures 692
i,iti
702 6.78
r:g 10.15
and Achieved Surveys
STAR Products
and Cooling Efficiency
and Shell Measures
lefrigerator and Freezer Recycling
782
2,490
5.03
1,2.o3
35.57
65
70
315 60 r 5.27
s.22
)o.zg
313 60
ss
rd Water Conversions
1,319
155 37 4.22
General Population Customer Surveys
Cadmus conducted two market characterization studies to build on previous evaluation findings and
supplement data from available regional resources, such as NEEA s Residential Building Stock
Assessment (RBSA). The purpose of this data collection was to help strengthen Avista's understanding
of:
Satu ration of key e nergy-efficiency measu res;
Key demographic and housing characteristics; and
Energy-use awareness, attitudes, and behaviors.
Our primary market research activity consisted of a multi-method survey that leveraged direct mail,
online web interface, and telephone calls to allow customer to complete the survey in the most
convenient way. The goal of these surveys was to characterize Avista's residential customers and allow
Avista to identify savings opportunities and possible new measure offerings. Cadmus also used this data
collection as a way to quantify nonparticipant customer spillover. We provide additional discussion on
this topic below.
a
a
a
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 27 ot'127
11
Table 7. Residential General Population Surveys Completed in2OL2 and 2013
Cadmus did not apply weights to survey frequencies during analysis. We based this decision on the
following rationale:
o Customers included in the general population survey sample frames were chosen at random
from Avista's entire residential population.
o The only screening was for completeness of customer contact information and removal of
customers targeted as part of other EM&V surveys conducted in 2011 and 2012.
o Cadmus concluded that there is no correlation between an inherent customer trait or
characteristic and the method of responding to the survey chosen.
Similar to the participant survey, the geographic distribution of survey respondents is clustered around
urban centers. Figure 2 provides the distribution of general population survey respondents.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 28 of 127
72
Figure 2. Geographic Distribution ol 2073 and 2014 General Population Survey Respondents
a
O.
a 'o
o
0a
o
o
'a't.iLi
oea@''o rll:I a
o
'e.
o
a
a.
,,i1,* .
t.
"o aa
a
a Oo
All participating customer and general population survey proportions reported below only include
feedback from respondents who could provide feedback-i.e., "don't know" and "refuse" responses are
not included in our reporting unless noted.
Status of Evaluation Recommendations
Avista retained Cadmus to perform annual process and impact evaluations of their residential program
portfolio beginning PY2010. These evaluation activities, findings, conclusions, and recommendations are
oo
a
Totll Comphtcd turvqt (2013 lnd 2014)
-_
t 147
Total Numbg of Houschold: (201,1 US Co:us)
O to 293
293 to 832
832 to 2,310
2,310 to 7,500
7 6m to 45-3m
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 29 of 127
13
articulated in the following reports: Avista 2010 Multi-Sector Process Evaluation Report and Avista 2011
Multi-Sector Process Evaluation Report.6
ln this evaluation effort, Cadmus reviewed the recommendations offered in these documents and
assessed to what degree Avista had adopted these recommendations (by the end of PY2013). As
indicated in Table 8, Avista made significant progress toward addressing these recommendations.
Table 8. Status of PY2010 and PY2011 Residential Process Recommendations
A complete summary of recommendations and activity for addressing these recommendations is
provided in Appendix A: Status of PY2010 and PY2011 Residential Evaluation Recommendations.
Progrom Porticipotion
Savings and lncentives
Table 9 provides the number of incentive-based measures and reported savings. The PY2012 and
PY2013 Avista lmpact Evaluation Reports explore the savings shown in Table 9 in detail.
'Avista 2o7o Multi-sector Process Evaluotion Report. Cadmus. 2011.
Avisto 2077 Multi-Sector Process Evaludtion Report. Cadmus,2Ot2,
74
Exhibit 3
Case Nos. AVU-E-14 AVU-G-'|4
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 30 ol 127
Table 9. PY2OL2-PY2O!3 Program Populations and Adjusted Gross Savings
!{ur1le_qs "4 Elg"tlg Plgglt.
ENERGYSTAR Homes 42 5,478
ENERGY STAR Products
High-Efficiency Equipment
Home Audit
Opower
Weatherization and Shell
7,233
5,906
477
857
3,670
898
L,O29
L3,204
555,076
Manufactured Home Duct Sealing 574 7,7t9 2,594 4L,978
0
928
73,497 i
421 t
9,091
251
239*
89,100 l
Electric-Only Programs
Second Refrigerator andfreezel Le!yg[q, _,_ t,,438 L,4L5 i, 1-,5q0 ,, O ]
s!mnres-!9q1,!m?t_sglnq: 111!,11 , !r6,!?9,i !s.37-3, __ _ ol
!pa9e a1d Watgr c91v,er1io11- 787 168 3,839 0 l
Total 452,346 678,593 68,747 lp}rs*Therm savings from the Opower program were very small and were not statistically significant.
A thorough discussion ofthe adjusted gross savings provided in Table 9 can be found in PY2OL2 - PY2013
impact evaluation reports.
Participation Trends
A review of Avista's residential portfolio over the past several years indicates several significant
transitions, specifically:
A sharp increase and subsequent decrease in participation in the ENERGY STAR Products and
Weatherization and Shell Programs (between 2008 and 2013);
Elimination of natural gas rebates in ldaho (November L,2Ot2l;
Reduction in the number of rebates offered for appliances (March 1, 2013); and
Commitment to developing and implementing new programs.
Cadmus combined historical participation data from PY2008 through PY2013 to assess participation in
Avista's rebate programs at the program level. These data, shown in Figure 3, clearly indicate increased
participation from PY2008 to PY2010, followed by a similarly abrupt decline in participation between
PY2011 and PY2013.
a
a
a
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 31 ol 127
15
Figure 3. Reported Number of Rebates by Avista-lmplemented Program: PY2008 - PY2013
3s,000
6 30,000
o
E 2s,ooo
& zo,ooo
b 15,000
5 1o,oooz s,ooo
0
600/o
t!40% *o20% !IUo%go-20% p
.EE-40% t)
tr-60% g
o-\OY" 4
I ENERGY STAR Products
Weatherization and Shell
I ENERGY STAR Homes
20L1 20L2 2073
r High-Efficiency Equipment
rSpace and Water Conversions
s-:1--) Percent Change Year-to-Year
--
I
=-il
8",-r
n
This trend runs against trends observed in appliance sales data in Washington and ldaho for the same
period. Overall sales generally dipped at the height of the recession and have since rebounded. Figure 4
shows population-normalized sales of several appliances in the ENERGY STAR Products Program (both
code and high-efficiency) as reported by the Association of Home Appliance Manufacturers (AHAM) for
Washington and ldaho from 2008 through 2013. This indicated that during this time period, a higher
percent of appliance sold were likely high-efficiency.
Figure 4. Population-Normalized AHAM Appliance Sales Data: 2008 - 2013
0.035
g
E o.osooE
b o.ozsr0p o.ozo
.g
E o.o1s
3 o.o1oo€! o.oosz
0.000
Clothes Washers Refrigerators
I 2008 r 2009 2010 r 2011
Freezers Dishwashers
.20L2 '.",20L3
16
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 32ot 127
Several explanations account for this decline in program participation. During interviews conducted to
inform the PY20t!,PY1O12, and PY2013 evaluations, Avista staff reported that a major driver of the
change was the expiration of many federal and state tax credits for energy-efficiency renovations and
high-efficiency appliances offered under the American Recovery and Reinvestment Act of 2009. Staff
reported these tax credits prompted increased participation in late 2009 and 2010, and beginning in
2011, participation slowed without that influence. This effect was particularly noticeable in the
Weatherization and Shell Program.
Another main cause of decline was the suspension of Avista's natural gas program in ldaho beginning
November t,zOtZ and plans to suspend natural gas programs filed in Washington. These changes led to
a dramatic change in the fuel composition of the residential programs between PY2O72 and PY2013.
Figure 5 provides a graphical depiction of this change. The few natural gas incentives paid in ldaho in
PY2013 were for applications submitted prior to the program change.
Figure 5. Distribution of Rebates from Avista-lmplemented Program Fuel Type: PY2OL2 - PY2013
?,!Lr: ,\ .;i /l
'-_l.l ,1
:\ l
14o
t,t44
'-^,-/(
Finally, in 2013 Avista also eliminated the ENERGY STAR appliance rebates (e.g., refrigerators, clothes
washers, etc.). A primary driver of this decision was increasingly high observed customer freeridership in
these measures and decreasing measureable gross savings. While Avista implemented this change in the
beginning of PY2013, Avista continued to process appliance rebates for projects installed within the
established 90-day grace period. This resulted in numerous units incented in the first half of 2013. Avista
took this approach to limit customer confusion and dissatisfaction around termination of the measure
offerings.
Not surprisingly, these changes had a large impact on the most common types of measures incented
through Avista's program. Table 10 shows the most common measures incented in PY2011 - PY2013 by
state, and the percent of rebates they represented.
..'-i. : \' I i 'r *^'.t : 1-- i " i/
i o*- '-
i I Electric
! I Natural Gas
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 33ot 127
77
Table 10. Most Common lncented Measures: PY2011 - PY2013
1 Refrigerator 15% Natural Gas Furnace- , - f N.trr.l G.r F,"n"n -,L2y.1 R.frig*.t*
Clothes Washer, Electric
H20
Clothes Washer, Natural
Gas water Heater
Clothes Washer -tt%Electric Water Heater
,rn i clothes washer -
Natural Gas Water Heater LL%
s I window Replacement 8% Variable Speed Motor
ldaho Measures
1 Refrieerator 16% Furnace--- -;-..
-.-;--.
23% Variable Speed Motor
i Clothes Washer, Electric Clothes Washer -i 2 r L4% Refrieerator 19o/o
-- i-- -l cLothes wast*-3 i Furnace t3% rr^*-:-rr,_.^-u^_.^_ 1 74% Refrigerator
Clothes Washer. Natural4 -'- -"-- 10% Variable Speed Motor tO% Air Source Heat Pump , !2% 'i
:. ._._ ._ Gas Water Heater
i , , Dishwasher, g% Clothes Washer - Natural g% Air Source Heat Pump 6%: El"Ulrydl$I91_ Il".t,.Ix:I
= Natural Gas Measure
* Avista eliminated refrigerator and clothes washer measures March 1, 2013, but allowed rebates for projects
completed in the 90-day grace period. This resulted in numerous rebates processed in the first half of the year.
Despite cancelling natural gas rebates in ldaho, a review of program tracking data indicates only a small
decrease in the percentage of Avista customers applying for multiple program rebates in a given
program year. Over the past three years, PY2011 - PY20L3, approximately one-quarter of participants
applied for more than one rebate. Table 11 shows the results, which exclude participants in the lighting,
refrigerator recycling, and behavior programs, as these are not rebate programs.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-l4
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 34 of 127
18
Table 11. Number of Measures lnstalled
One t4,O62 77o/o 8,953 78Yo 2,8L3 74%
Two 3,L27 t7% 1,936 t7% 815 - 2L%
Three 784 4% 424 4% 153 4%
Four ' 172 7% I 91 7/o ' 27
It is not uncommon for customers to participate multiple times over several years, although, as
indicated in Table 12, this is becoming less common. This downtick is likely the result of more limited
rebate offerings, particularly in ldaho, than in previous years.
i-?0f1!3tf,p3lt'th{!9lI[atgl:910 , lL .
2012 participants that qgltrcfglgg,, :0j+ -: ]91_
?013 partilieglts tha! !r ItLcl'p9leq in 201? - i _ !v:
2013 participants that participated in 2011 and 2012 t%
Customer intentions expressed in PY2013 and PY2012 participant surveys show that the decline is not
likely due to lack of customer interest. As indicated in Figure 6, when asked if they thought they would
apply for additional rebates in the future, more than half of PY2013 respondents in every program
answered in the affirmative. Further, we see a strong increase in the respondent interest in participation
compared to results from PY2012 across all programs.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 35ol 127
19
Figure 5. Customer lnterest in Repeat Program Participation
80%
70%
60%
50%
40%
30%
20%
LO%
o%
ENERGY STAR Heating and Space and Water Heating Weatherization Appliance
Products Cooling Water and Shell Recycling
Efficiency Conversions Measures
r 2012 (n=335) I 2013 (n=309)
The decline in rebate program participation is significant, but review of annual reports from other
utilities in the region-Pacific Power in Washington, and Rocky Mountain Power and ldaho Power
Company in ldaho-indicate similar reductions in participation in their electric rebate programs with
comparable measure offerings.
Figure 7 provides the number of reported rebates, by category, from year to year. All three utilities have
experienced net negative growth, without exception, in the number of participants in these measure
categories since 2011.
Figure 7. Participation Trends Among in Rebate Programs among Regional Utilities: PY2008 - PY2012
20,000
f; rs,oooltoEt 1o,ooo
oltE s,oooz
0 6lorlOlHlNOlOldlFllFlololololoNININiNIN
Rocky Mountain
Power (lD)
6lorloldlNololdldldololololoNINIdININ
ldaho Power
Company (lD)
6loilolrrlNOlOlHlFllFlololololoNININININ
6lo)lolFrlNOlOlFllHldololololoNININININ
Pacific Power (WA)Avista (lD & WA)
I HVAC (and Water Heating) r Appliances ! Weatherization and Shell
20
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 36 of 127
While participation in Avista's rebate programs has steadily declined for the last three years, Avista has
maintained its commitment to third-party implemented programs-such as Second Refrigerator and
Freezer Recycling-and regional programs such as Simple Steps, Smart Savings. Due to this support,
participation in these programs has generally remained level or increased. ln addition, in PY2OL2 -
PY2013 Avista successfully implemented two pilot programs and a large, fully developed behavior
change program. Figure 8 provides a summary of customer participation in these programs. For some
programs, participation is shown in "100s" as participation in these programs is significantly higher than
others.
Figure 8. Reported Number of Rebates by Non-Avista-lmplemented Program: PY2010 - PY2013
o
=.Eo
=o
o3
E
=z
12,000
10,000
8,000
5,000
4,000
2,OOO
0 2010 20tL
r Home Audit
Residential Behavior (100s)
r Simple Steps, Smart Savings (100s)
I Manufactured Home Duct Sealing
I Second Refrigerator and Freezer Recycling
A possible reason for growth in the Simple Steps, Smart Savings Program is the recent introduction of
two additional measures: energy-efficient showerheads (introduced inPY2OLZ); and LEDs (introduced in
PY2013). Table 13 provides additional detail on uptake ofthese new measures.
0
31,5!7 '
o%1428,6781 LOO%
ry hrr* i --eit l
Another possible reason is the increase in the number of participating locations. According to invoice
materials, 92 locations participated inPY2OL2 compared to 125 in PY2013. These additional locations
give Avista customer greater access to incented measures.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-l4
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 37 of 127
Table 13. Simple Steps, Smart Savings Measures lncentives in PY2012 - PY2013
27
Progrom Design, Monogement, ond lmplementqtion
This section discusses Cadmus' observations regarding design of Avista's residential programs. These
observations focused on program definition and organization, logic, and implementation approach.
Overview
Overall, we found Avista's the residential program designs work well and are generally well-
documented, primarily in the PY2012 and PY2013 DSM Business Plans. Further, we found Avista
management and implementation organization staff to be knowledgeable about the programs and
invested in their ongoing success. ln general, the PY2012 and PY2013 the programs operated smoothly,
with few significant issues.
However, Cadmus did find one persistent program design issue. First noted in Cadmus'2010 residential
program process evaluation,T the naming convention of programs composing the residential portfolio is
somewhat inconsistent across Avista Business Plans, marketing materials, and internal documents. ln
reviewing materials, it became clear that programs are often referred to with different namet and are
organized differently within the portfolio. Table 14 identifies several programs as examples.
New Construction / Home lmprovement I High efficiency Equipment
WeatherizationHome lmprovementFuel-Efficienry .
ENERGY STAR Homes ENERGY STAR Homes ENERGY STAR / ECO-Rated Homes
Program Logic
Camus developed the logic model provided as Figure 9 to articulate the logic behind the residential
programs included in this evaluation.
7 Avisto 2077 Multi-sector Process Evoluotion Report. Cadmus. 2012.
60
bsEU0'69pe Conversion from Electric
Table 14. Example of Residential Program Naming Convention
22
Exhibit 3
Case Nos. AVU-E-l4 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 38 ot 127
:= eNE,I, 1FEYgb
'r= 99< (r;:gP
+Ed.I (!(Y))O o{ oE
8FEZ ciEO .FU)
EEY
U)
(n
IrItlr
' lE'lr
iliiT
r{ll}ffi{:::-$iii
t
ft
I
II
I
2t
ir
I"lir
t
tIr
itr
I
uU
$
T
f
T
I
I
t
T
.iI
I
il
II
il
iT
TT
Ef
Ii
ir
$[
$T
#E
irtr
gT
E
I
T
b
Et
o1'o
=g'&oJ
Et!
d0o
o.
6.E
troB
oe,o
.9
oio
,g!r
l{E
l[[
fr
IE
I[rfi,SEI
I m plementotion Approache s
The residential portfolio includes programs with Avista administers, programs with third-party
implementers, and programs operated as partnerships. This section summarizes our observations
regarding Avista's implementation decisions for each residential program.
Avista residential programs are implemented both internally and with the assistance of several third-
party organizations. Table 15 provides a summary.
ENERGY STAR Products
High+mciency E;ripr..t -
I H.re Ardit
-- l
Manufactured Home Duct Sealing
Residential Behavior Mgmt. QAv/qg, and invoice PaYment
Weatherization and Shell All implementation activities
I Electric-Only ProgramsI EteGf[tc-vnly ragBJdat
l_'': ---; . _;
I Second Refrigerator and Freezer
I h-----r:--Mgmt. QA/QC, and invoice payment
Simple Steps, Smart Savings CLEAResult
Table 15. Avista Residential Program lmplementation Approach
Natural Gas and Elecuic Protrams
Avista and NEEA i Mgmt., marketing, QA/QC, and rebate payment i
^r* -
_-] All implementation activities
i
-lM*i.i*l P"rt*"-- i
I tanilc
i Space and Water Conversions j All implementation activities
Staffing
Despite these implementation partnerships, over the past several years, Avista has continued to invest
in the implementation and management of its energy-efficiency portfolio. A review of Avista DSM labor
projections articulated in the 2012 and 2013 DSM Business Plans indicates a generally increasing
number of full-time-equivalent (FTE) staff dedicated to program implementation and management
activities (Figure 10).
Exhibit 3
Case Nos. AVU-E-I4 AVU-G-I4
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 40 oi 127
24
Figure 10. Avista DSM Labor Projections: PY2008 - PY2013
35.0
g 30.0trg
.E zs.o
ETr
E 20.0
-9f rs.o
t!o
.E 10.0
E
'l 5.0
0.0
Also reflected in this staffing increase is the addition of a third and fourth Avista program manager in
2012. Avista added these program managers for the additional work associated with the Residential
Behavior and Manufactured Home Duct Sealing Programs. Both staff had previous experience with
Avista's residential energy-efficiency programs. lnterviews with Avista staff indicate staffing levels during
PY2OL2 and PY2013 were adequate and no significant implementation staffing issues arose.
The four program managers have responsibilities beyond residential program management. To support
these program managers, a team of staff contributed to day-to-day program operations, including
customer outreach, application review and processing, and data management. ln addition to oversight,
the program managers also conduct regular quality-assurance tasks. For example, the program manager
responsible for Simple Steps, Smart Savings regularly visited participating retail stores to ensure correct
prices and correct display of point-of-purchase signage.
As Cadmus did not study Avista's costs in administering these programs, this report does not address
their relative efficiency. However, following a recommendation in the PY2011 process evaluation report,
Avista and Cadmus staff discussed the possible benefits of contracting elements of the program
implementation (e.9., rebate processing). The conversations, while focused, identified no compelling
reasons for Avista to consider transferring additional program elements to third-parties at that time.
Customer lnteraction
Feedback from Avista staff indicates customer satisfaction is a high priority for the organization, and
energy-efficiency programs are viewed as a powerful method to engage with customers. To ensure
customer satisfaction, Avista staff take care in program marketing to limit messaging that might confuse
customers-such as why natural gas rebates are available in Washington but not ldaho-and to process
rebate applications promptly-a common area for customer dissatisfaction in utility rebate programs.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 41 ol 127
A review of program data indicates Avista has a strong record of processing rebates within days of
receipt, although in PY2013 processing time slipped slightly (Table 15). This increased processing time is
likely related to the elimination of the appliance rebates, leaving only the more complicated rebate
applications that may take longer to process.
The increase in processing time shown in Table 15, two days on average in PY2013 compared to less
than a day in PY2012 and PY2011, is also primarily the result of a few applications with processing times
far outside the normal range (e.g., greater than 100 days) skewed the average processing time upward.
Many of these database entries contain notes indicating issues with customer application completeness.
To achieve these quick application reviews, Avista implements a structured review process supported by
several internal staff. Review staff also regularly follow up directly with customers via telephone calls in
the evening, when customers are likely to be home, to address application issues directly. ln addition, an
increased percent of participants are submitting their application papenuork in electronic format online
(Table 17).
Table 17. Percent of Applications Submitted ln Electronic Format Online by Program
To inform both the impact and process assessments, Cadmus conducted desk reviews of more than two
hundred applications in 2013 and 2014. Table 18 provides a summary.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 42ot 127
Table 16. Rebate Processing Times: PY2011 - PY2013
I Five or more days
Space and Water Conversions | 5%
26
Table 18. Summary of Cadmus Desk Reviews
ENERGY STAR Homes
ENERGY STAR PToducts
rrore"t;il;r.reil-
(H E eq u i pm e nt, weathe rizati on, a n d conve rsio n )
While application processing is generally quick, Cadmus' review of original application materials from
PY20L2 and PY2013 identified some issues with completeness of documentation. Table 19 lists elements
that were missing in original application materials, as identified in our application review. No issues
were identified in ENERGY STAR Home applications.
ENERGY STAR Products L
Home lqp1gygqgq1t
PY2O13 Review
ENERGY STAR Products i , ',
len" lrPreve-lle"J -- -_-a-
lnterna! Communication
During the PY2011 process evaluation effort, Cadmus identified different perspectives among Avista
staff around program planning and goal setting. ln the PY2011 report, we noted: "progrom manogers
depicted the Plonning, Policy, ond Anolysis (PPA) teom os the driver of the plonning processes, while the
PPA team noted progrom plonning wos the responsibility of the progrom monogers. This disconnect
oppeared to result in unmet expectdtions for both teoms, ond moy hove impeded effective
colloborotion."
To address this and other collaboration issues, between PY20t2 and PY2013, Avista invested heavily in a
self-evaluation of internal communication protocols (primarily between engineers, account executives,
program managers, and PPA staff), and staff roles and responsibilities. To facilitate this assessment,
Avista retained the services of Milepost Consulting, a third-party consulting firm specializing in process
improvement. Cadmus was not directly involved in these activities.
According to Avista staff, this self-evaluation effort has had a limited impact in addressing the issues,
and communication and collaboration between groups continues to present challenges. Further, Avista
initiated a reorganization of the DSM team in April 2014, which placed program implementation and
PPA staff under one common Senior Director. Cadmus strongly supports Avista's commitment to
internal process improvements and decision to adjust the internal organization.
L8_
1P,
102
23
t4
Exhibit 3
Case Nos. AVU-E-14 AVU-G-'|4
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 43ol 127
Table 19. Summary of Missing Application Elements
PYzOLz Review
Third-Pafi Program lmplementation
Avista uses third-party implementation contractors for four programs, not including the Home Audit
Program: (1) Manufactured Home Duct Sealing; (2) Residential Behavior; (3) Second Refrigerator and
Freezer Rerycling; and (4) Simple Steps, Smart Savings. We provide a summary of these arrangements
and an assessment of their effectiveness in the Effectiveness of lmplementers section, below.
Eflective n e ss oI I m pl e mente rs
Using third-party implementers presents advantages and disadvantages. Generally, utilities maintain
direct implementation of programs requiring intimate knowledge of unique customers (e.g., large
commercial and industrial customers). Programs benefitting from a uniform approach involve national
accounts, or require certain market expertise available from a third-party firm. Research conducted for
this-and previous-Avista evaluation efforts leads us to conclude that Avista has succeeded in
identifying which programs are most suitable for third-pafi contracting and partnering.
The PY2011 evaluation report provides the results of detail interviews conducted with implementation
staff at JACO and CLEAResult. As few changes have been made to these programs since these interviews
took place in late spring 2OL2, we focused our evaluation efforts on Opower. Opower implements the
Residential Behavior Program, which began in June 2013.
Opower
Opower is a publicly held (as of April 4,2OL4l software-as-a-service company that partners with utilities
to implement behavior-change programs. Based in Arlington, Virginia, Opower has been involved in the
energy-efficiency space since 2007 and currently partners with nearly 100 utilities in the United States
and abroad.s ln April 2014, Cadmus staff interviewed the Opower sales and engagement manager
responsible for Avista's program.
Residential Behavior Program Description
The Residential Behavior Program encourages electric customers to implement free or low-cost
measures and adopt energy use practices and behaviors that reduce electric consumption. Program
participants were selected by Avista (with support from Opower and Cadmus) and receive a Home
Energy Report from Opower in the mail. All customer calls are addressed by Avista's call center. The
Home Energy Reports include the following information:
o Comparisons of a custome/s usage in the current year to consumption in the same months in
the previous year.
. Comparison of a custome/s consumption to consumption of other, comparable customers in
the same geographical area. This is known as the "Neighbor Comparison."
o Tips about how to save energy and reduce demand during peak times. These tips include:
t opower. April 8, 2014. http://opower.com/company.
28
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page M of 127
. General conservation tips such as turning down the thermostat, turning off lights,
shortening showers, etc.
t Low-cost energy-efficiency tips, such as replacing incandescent bulbs with CFLs, installing
weather stripping, and using power strips.
t Tips about ways to reduce peak loads during peak load season and shift energy use to off-
peak periods.
. lnformation on other Avista residential programs.
No financial incentives are provided through this program.
According to the program theory by educating customers about their energy use and conservation
strategies, customers will gain knowledge to increase their energy efficienry and achieve cost savings. ln
addition, customers will become more engaged with Avista.
Currently Opower reports only electric savings to Avista, although some customers may also have
natural gas service and may take actions to reduce their use of this fuel as well. Avista and Opower may
take steps to quantify these savings in the future.
Residential Behavior Program lmplementation
Avista implemented this program using an experimental research design with random assignment of
customers eligible for the program to treatment and control groups. From their residential customer
population, Avista, with support from Opower and Cadmus, selected approximately 70,000 customers
for inclusion in a treatment group and 13,000 customers in two, state-specific, control groups (a total of
26,000 customers). Avista did not consider natural gas-only customers. Based on initial cost-
effectiveness analysis for program planning, Avista set a minimum energy consumption threshold of
18,000 kWh per year for targeted households. ln order to fully populate the participant and control
groups in both Washington and ldaho, Avista reduced this threshold to approximately 15,000 kWh as
the program was deployed.
Treatment group customers received Home Energy Reports beginning in June 2013 and then according
to the schedule provided in Table 20. To use implementation resources such as printing and call center
staff as efficiently as possible, Opower mails reports in batches staggered throughout the month.
Control group customers did not receive Home Energy Reports and were not informed that they
belonged to the control group. Opower uses this general approach for most of the programs it
implements.
Table 20. Home Energy Report Deliver Schedule
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 45 ol 127
Opower works with Avista's billing department to access customer billing data. Using these data,
Opower staff quantify program kWh savings. Cadmus reviewed the saving estimates as part of the
PY2013 impact assessment and performed an independent billing analysis to determine gas and electric
savings.e
According to Opower implementation staff, the Residential Behavior Program has operated as
anticipated since inception with only minor challenges. Staff report a very strong relationship with
Avista, noting the Avista team is:. "super responsive, very polite, ond very nice to deol with...overoll it's
one of the heolth[iest] client relotionships we hove." The only challenge noted has been with the
customer usage data used to populate the Home Energy Reports, but both Opower and Avista are aware
of the issue and are working to streamline the process.
Participant feedback to the program has been positive. While data were not readily available for this
evaluation, implementation staff estimated that-so far-less than one percent of participants have
contacted Avista expressing dissatisfaction in the program, and opt-out rates are lower than expected.
Only 1.0% of customers in Washington and t.L% of customers in ldaho have requested to be removed
from program mailings as of April 2014.
Future of the Residential Behavior Program
Given the success of the program, in terms of both implementation and achieved energy savings, Avista
and Opower have discussed the possibility of either expanding the program or fine-tuning by targeting
specific customer groups. No firm plans are in place, but discussions around this topic are scheduled for
later in spring 2014 in order to consider results of Cadmus' impact evaluation of the program. Given that
Avista has already included all cost-effective customers in their target population for this program,
future opportunities for expansion may be limited.
Doto Trocking
For each residential program evaluated, Avista or the program implementer provided Cadmus with
tracking data. Tracking data were contained in five separate files:
o Avista's internal, multi-program tracking database;
o Manufactured Home Duct Sealing tracking spreadsheets;
o JACO tracking database;
. Opower tracking database; and
o Simple Steps, Smart Savings invoice material.
Cadmus examined each dataset to:
' Avisto 2072-2073 Woshington Electric lmpoct Evoluotion Report. Cadmus.2Ot4.
Avisto 2072-2073 ldoho Electric lmpoct Evoluotion Report. Cadmus.2Ot4.
30
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 46of 127
. Determine data fields tracked;
o lnform process and impact evaluation activities; and
o Assess the data-tracking processes' effectiveness.
The assessment also sought to identifo potential evaluability barriers presented by current tracking
processes,
Data Tracking Summary
Avista's lnternol Multi-Program Tracking Dotobose
The tracking database included participant, measure-level data for the following programs:
ENERGY STAR Homes;
ENERGY STAR Products;
High-Efficiency Equipment;
o Weatherization and Shell; and
. Space and Water Conversions.
The internal, multi-program database serves as the electronic repository for customer data collected
from application forms, including data for programs Avista implements internally. The two annual
extracts provided for this evaluation effort contained 38 variables, constituting six kinds of information.
Table 21 summarizes these data.
Customer lnformation Number / Text "5tote,CUSTOMER_NME, Home Sq Ftg, Yeor Built"
iiiri"iiv notirs, New R iitie, tnstoll oote"lncented Equipment lnformation Date/Number/Text "Cost,
Measure / Rebate Quantities Number "Number of Rebotes"
Measure and Program Designation Number / Text "Morketing Meosure Type, Marketing Meosure Desc"
Pa-ynrent and Savings Number ;A"i.t" A^ount, Est KWH Soved, Est tnerms iovZa'
a
a
a
a
Processing Date-Stamps and Notes Date / Text "App Rcvd Date, Poyment Processed Date"
We also know from ad hoc requests that Avista tracks several other data in addition to the items
outlined above. These variables include a "Do Not Solicit" customer flag and several customer contact
and billing information fields with additional detail and formatting.
Manufoctured Home Duct Sealing Trocking Spreodsheets
The Manufactured Home Duct Sealing data extract reviewed in this evaluation contained three quarterly
summaries. Tracking data contained 36 fields, including: customer address; Avista account information;
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 47 ot'127
Table 21. Avista lnternal Tracking Database Fields
31
duct-sealing services performed; and energy savings estimates. We understand from conversations with
program staff that information on each job are provided in bulk by UCONS, the implementer and
additional fields are then added by Avista staff during the QC process.
IACO Trocking Dotobase
JACO tracks data on participating customers, their pick-up orders, and refrigerators and freezers
recycled through the program. These data are provided in three separate, integrated spreadsheets,
allowing comprehensive tracking of customers' and units' movements through the program.
Through our experience evaluating Avista's Second Refrigerator and Freezer Recycling program and
other similar utility-sponsored appliance recycling programs implemented by JACO, we know these data
files are consistent in content and format with JACO's standard program tracking. While these data are
detailed, providing extensive information on the customer, pick-up, and equipment recycled, Cadmus
noted the absence of an Avista customer account number. JACO assigned customers their own unique
customer identification numbers.to This made it difficult to match customers participating in this
program to other program tracking databases.
Opower Trocking Dotobose
Opower, the Residential Behavior program implementer, provided the program tracking data we
reviewed for this program. The tracking database contained only 10 fields for each participating
customer, listed in Table22.
Table22. Opower Data Tracking Fields
"opower_customer_id"_-'1rt,lity-.r""r*-id'
@
"service-address" :
"recipient_status"
"opt=o-ut_clate"
"inactive_date"
"incl ude_in_test_a na lysis"
"deployment_wave"
"firstJenerated_date"J
Through our experience evaluating other residential behavior programs implemented by Opower, we
know these data files are consistent in content and format with their standard program tracking.
10 Customers sign up for the program, either online via Avista's website or by calling JACO's toll-free number. They
are asked a few questions to verify eligibility, they must be Avista electric customers, and their refrigerator or
freezer must meet certain criteria to participate.
L
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 48 ot 127
32
However, unlike tracking data from other third-party program implementers, this dataset includes
Avista customer account number (utility-customer-id).
Simple Steps, Smart Savings lnvoice Material
Cadmus received data on the Simple Steps, Smart Savings Program. This program tracks monthly
reporting from CLEAResult. ln interviews conducted to inform both this and the PY2011 evaluation,
Avista and CLEAResult staff noted monthly reporting for this program often involved delays and
adjustments, caused by difficulties in obtaining sales data from retailers in a timely manner. CLEAResult
monthly invoices contained detailed data at the measure level, reporting adjustments to previous
months, and current monthly sales at each participating retailer by Stock Keeping Unit code (SKU). Data
reviewed for this evaluation contained slightly different fields, but both provided information on:
. Participating retailer (e.g., name and location);
o Measures (e.g., manufacturer, type, SKU, watts/GPM, etc.);
o Sales and sales adjustments; and
o Reporting period.
Planned Changes in Avista Data Tracking
ln addition to maintaining the internal tracking database discussed above, Avista is currently engaged in
a large, multi-year transition to an advanced customer care and billing system, supported by Oracleo.
This transition has been in progress since 2012. ln July 2014, Avista hopes to begin moving some aspects
of its energy-efficiency program tracking to this new system. Anticipated benefits with this new system
include improved access to complete customer account information, enhanced market segmentation
tools, and targeted marketing campaigns.
Marketing ond Outreach
Marketing Approach
Avista develops, executes, and oversees the marketing efforts to promote its residential rebate
programs in Washington and ldaho. These efforts include paid media, social media, earned media, direct
mail, website, and broad-based awareness building through the "When it comes to energy efficiency,
every little bit odds up" (Every Little Bit/ campaign, along with the Efficiency Motters campaign. Most of
the outreach tactics include general promotion of the residential rebates, with individual measure or
program promotion as needed. Additionally, some program implementers supplement Avista's
marketing through their own turnkey efforts. Avista's energy-efficiency marketing efforts are also
coordinated with regional efforts.
Cadmus conducted a review of Avista's residential energy-efficiency rebate program marketing efforts
to:
33
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 49 ot 127
. Gain an understanding of PY2012 and PY2013 energy-efficiency and program marketing
strategies and processes;
o Understand customer response and gauge effectiveness of marketing efforts; and
o ldentify gaps and/or opportunities for consideration in future marketing efforts.
As part of this effort, Cadmus conducted a marketing materials review. We also reviewed marketing-
related survey results and Avista marketing staff interview findings.
Marketing Objectives and Strategies
As found through review of the 2013 marketing plan and as supported through the interview with Avista
marketing staff, the overarching outreach objectives are to increase awareness of and participation in
Avista's energy-efficiency programs for residential customers. The outreach strategy is to use varied
media to highlight customer success stories and communicate program benefits through engaging and
interactive promotions and partnerships. Avista's DSM plan also indicates that residential programs
have a strong presence and coordination with regional efforts, such as those offered by NEEA.
ln our interview with Avista's key marketing staff, we discussed energy efficiency marketing successes
and challenges in the PY2013 year. Overall, Avista staff reported the marketing efforts had been
successful-specifically the online Every Little Bit and Efficiency Motters campaigns and high-performing
targeted online advertisements. Staff indicated the crossover between Washington and ldaho (and
offerings, based on fuel type and regulations) continues to prove challenging with regard to messaging
and delivery of mass media. Staff reported they believe the Every Little Bit and Efficiency Motters
campaigns are helping to increase broad-based reach to audiences without the use of mass media. ln
looking forward, staff indicated a need to enhance energy-efficiency awareness and participation
through deeper and more meaningful customer engagement. Avista staff hope to learn more about
customer motivators and how staff can increase customer engagement along the path to participation
in residential energy-efficiency programs.
Planning and Processes
Avista staff conducts the planning, design, and execution of the residential rebate program marketing
efforts. As indicated in the PY2012 and PY2013 DSM plans, there is an internal collaborative process to
develop general energy-efficiency marketing and promotions. This process incorporates feedback from
the Energy Solutions, Services Development and Marketing and PPA teams. Some of the turn-key
programs, such as the Second Refrigerator and Freezer Recycling Program, include supplemental
marketing as part of their program design and implementation plans.
Avista's marketing staff uses the Avista Design System Guidelines to ensure that energy-efficiency
marketing and outreach materials deliver a consistent look, feel, and message. The guidelines address
items such as logos, color palettes, and fonts, and give an overview of applications, with examples of
properly branded materials and collateral. All PY2012 and PY2013 general energy-efficiency marketing
materials appear to be aligned with the guidelines. The Every Little Bit and Efficiency Motters campaigns
34
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 5O ol 127
and Online Energy Advisor tool present slightly varied creative assets, although generally appear to
follow the brand guidelines (i.e., fonts, logos, etc.).
Target Audience and Customer Motivators
The target audience for Avista's residential rebate programs is general, and Avista has not specifically
segmented customers or targeted outreach efforts. However, based on interviews with Avista staff, the
marketing strategy uses a variety of outreach channels to reach a mix of demographics. For example,
print ads are used to reach an older customer audience, while online advertisements are aimed at a
younger demographic. Although segmentation efforts have been limited to date, Avista staff hopes to
have a better grasp of customer segments and preferences in the future.
Avista reported conducting a residential customer market research survey in 2013 with 400 customers
in both Washington and ldaho. The purpose of the research was to gauge awareness of Avista's
programs and to gain insights to key motivators and messages. Avista will use these data to develop its
PY2OL4 marketing and messaging strategies.
The participant surveys conducted by Cadmus also explored motivations for program participation. The
most common responses from PY2012 and PY2013 are provided in Figure 11. The most commonly
reported deciding factors were old equipment working poorly (25%, up from L2%in2OL2l and old
equipment not working (22% up f rom 78% in 2012). The two responses totaled 48% in 2013. Responses
reflect the changing composition of residential rebate offerings. The response "like the appearance of
the new item more" is a common response amount customers who received a rebate for an energy-
efficient appliance-which were eliminated in PY2013.
Figure 11. Most Commonly Reported Measure Purchase and lnstallation Motivations
Old equipment working poorly
Old equipment didn't work
Wanted to save energy
Wanted to reduce energy costs
Recommendation of dealer/retailer
Liked the appearance of the new item more
o% 5% t0% Ls%
I2013 (n=2511 t2Ot2 (n=a73)
35
Exhibit 3
Case Nos. AVU-E-l4 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 51 ot 127
Outreach Channels
Avista conducts residential energy-efficiency marketing through a variety of channels. ln addition to the
general energy-efficiency marketing tactics outlined below, Avista conducts broad-based awareness
efforts through its Every Little Brt campaign, as described in the following section. Besides the Efficiency
Motters campaign (which are implemented in partnership with KREM 2, a CBS affiliates), there are no
mass media or cross-cutting promotional efforts related directly to program offerings, to avoid potential
customer confusion across state lines.ll Notable outreach tactics used in PY2012 and PY2013 include:
o Paid media: print and online (targeted SEO) banner advertisements;
o Social media: Facebook, specifically for campaign and ticket giveaway;
o Earned media: local public relations as available;
e Direct mail and bill inserts: general and (targeted) program-specific;
o Newsletters and e-mail blasts: general outreach;
o Website: website (avistautilities.com) was built in 2OL2; and
o Vendor outreach meetings: general overview about programs, application process, project
qualifications and customer eligibility.
Every Little Bit and Efficiency Motters Campaigns
The Every Little 8ft campaign launched in 2OO7 and was informed by findings from market research
efforts that gauged customer awareness, willingness to participate, and barriers to participation. The
broad-based, multi-media awareness campaign was designed to increase customer engagement and
drive awareness of Avista's energy-efficiency program offerings. Over the years, the campaign has used
multiple channels, including website, web banners, print and broadcast outreach (radio and television),
print material (brochures, signage, etc.), outdoor billboards, social media, and community events. The
objective of the campaign is to educate and inform customers about general energy efficiency programs,
with the goal of driving participation. The call-to-action drives customers to Avista's campaign website
(www.everylittlebit.com) to take advantage of energy saving programs from Avista.
During subsequent years, the program design shifted to become progressively more specific. Most
recently, KREM 2's Project Green, Toyota and Avista have teamed up in support of energy efficiency, and
initiated the Efficiency Matters campaign. Through this campaign, customers entered to win a Toyota
Prius by pledging a commitment to energy efficiency. Objectives of the most recent campaign were to:
o lncrease awareness of and participation in Avista's energy conservation measures and rebate
programs;
o lncrease traffic to www.everylittlebit.com;
" Avista also partnered with the lnlonder newspaper and ACE Hardware to promote its Home Energy Advisor
online audit tool.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 52oi 127
35
. lncrease traffic and "likes" to the Efficiency Motters Facebook page; and
o Allows people to receive ongoing energy-efficiency tips.
Through its partnership with KREM TV and Toyota, Avista's campaign garnered more than 103,000
entries in 2013, with 4,159 people searching for the Every Little Bit keyword. There were 55,907 total
entries the previous year.
Materials and Messaging
Cadmus reviewed all residential energy-efficiency marketing materials provided by Avista. Overall, the
general marketing materials present a consistent look and feel, and follow the Avista Design System
Guidelines (e.g., fonts, colors, layout, and applications). Materials typically include the Avista logo
(appropriately) and a call-to-action, which is usually one of Avista's websites (or campaign URL). The
online advertisements direct customers to the program webpage, which serves as a portal for customer
engagement, education and interaction and provides links to rebates and tips. Several ofthe general
marketing materials also include program-appropriate imagery which may help customers understand
and relate to the promoted offerings.
Through our review of PY2072 and PY2013 materials, we found there are several uniform resource
locators (URLs) included in the collateral, and some items including more than one URL (e.g.,
www.everylittlebit.com, www.everylittlebit.comffindrebates, www.avistautilities/resrebates).
lnconsistent use or use of more than one URL may distract customers and possibly cause confusion.
While the materials reviewed focused primarily on the general residential rebate marketing materials,
Cadmus also reviewed Every Little Bit and Efficiency Matters campaign outreach materials and Avista's
energy-efficiency web pages, and conducted a high-level review of the Online Energy Advisor materials
as a point of reference, Based on this cursory overview of the suite of programs and platforms, Cadmus
found that there are varied creative assets across the channels and platforms. While the general energy-
efficiency promotional materials present a look and feel consistent with the brand guidelines,the Every
Little Bit and Efficiency Moffers campaigns and Online Energy Advisor platforms leverage additional
assets. For example, lhe Every Little Bit landing page (www.everylittlebit.com) also includes assets from
the Online Energy Advisor personas (with the "shield" creative) and creative developed by a third-pafi
implementer.
Marketing Execution and Measurement
Avista tracks metrics for its individual campaigns and ties results back to awareness and website traffic.
ln PY2013, Avista staff reported tracking online advertisements (click-through ratesl, Every Little Bit and
Efficiency Matters campaign metrics (participants and traffic), estimated impressions through paid
media and response to direct mail (as applicable).
Sources of Participant Awareness
To help assess the effectiveness of Avista's and the implementer's marketing; Cadmus asked
participants how they heard ofthe program in which they participated. Respondents cited a variety of
i7
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 53 ot'127
sources of program awareness. Figure 12 lists the top ways respondents said they first heard about the
program in both the PY2012 and PY2013 surveys.
PY2013 respondents who could provide an answer reported hearing about the program through their
contractor (28%), with other responses fairly evenly distributed across information from electric or gas
bill (15%), word of mouth (14%), and the Avista website (LZYol. When Cadmus compared 2OL2 and 2OL3
findings, a few key differences emerged:
o More respondents heard aboutthe program from a contractor in 2013 (L7%in2OL2,Z8o/oin
2013).
o Fewer respondents heard about the program from a retailer/distributor in 2013 (15%in2OL2,
5%in20L3l.
o Fewer respondents heard about the program from an Avista representative in 2013 (LL%in
2012,7% in 2013).
Figure 12 provides additional customer responses.
Figure 12. Most Commonly Reported Ways Participants First Heard About the Program
Contractor
lnformation with my electric or gas bill
Fa m ily/f riends/word-of-mouth
Avista Website
Avista representative
Retailer/Dealer
o% 5% to% t5%
r2013(n=323l, r2072(n=597)
Not surprisingly, the ways participating customers first learned of the Avista rebates differs by program.
For example, we would expect customers seeking HVAC and weatherization rebates heard of the
program from their contractor, while ENERGY STAR Products customers more commonly heard of the
rebate from a retailer. Figure 13 provides the most common responses by program.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 54of 127
38
Figure 13. Most Commonly Reported Ways Participants First Heard About the Program by Program
a0
.gU
Ioc,
6sttl
dx:
uo.=ooI
;
.9
o
co(J
Icoo
U
ufioo
CL
0,Ic.q
CLCL
o
E
=
@
E
=
o!.gEo(J
E
F6
oUoo6o"
oU
oo.c
oo
I
(,
ccUzU
NHoN
mdoN
Utility bill insert (n=38)
Utility bill insert (n=25)
Contractor (n=33)
Contractor (n=21)
Utility bill insert (n=21)
Contractor (n=10)
Contractor (n=7)
Avista website (n=9)
Contractor (n=48)
Contractor (n=29)
Retailer / dealer (n=35)
Retailer / dealer (n=16)
NHoN
(ndoN
NdoN
(o
oN
NioN
(oHod
NdoN
mdoN
NdoN
moN
Avista Customer Awareness of Energy-Efficiency Rebates
More than half of Avista's residential customers report being aware Avista offers rebates for energy-
saving equipment and weatherization improvements when asked as part of the Avista general
population studies. lndicated in Figure 14,63% of customer surveys in2OL2 and54Yo of customers
surveyed in 2013 reported being aware of Avista rebates (prior to completing the survey). The decrease
in awareness reported in 2013 compared to 2OL2 may reflect the reduction in rebate offerings in ldaho
as well as the challenges Avista faced in marketing dissimilar measure offerings across the two states.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 55ol 127
Figure 14. Avista General-Population Customer Awareness
70%
60%
50%
40%
30%
20%
to%
Olo
I Aware of Avista rebates
(prior to taking survey)
I Not aware of Avista rebates
2012 (n=1,019)2013 (n=1,058)
Customers who reported being aware of Avista rebates indicated that information in their utility bill was
the most common way they learned of the measure offerings (38% in 20t2 and 43%in 20L31. Word of
mouth (13% and L4Yo), the Avista website (LL% and 9%o) and W advertisements (11% and 8%) were the
next-most-common responses, although feedback was diverse. Figure 15 provides additional detail.
Figure 15. Source of General-Population Customer Awareness
lnformation with electric or gas bill
Family / friends / word-of-mouth
Avista website
TV
Newspaper
Contractor
Radio
Avista representative
Magazine
Social media (Facebook, Twitter, etc.)
Event
Billboard
Other website
Something else
r 2013 (n=8551 .2012 (n=1,073)
o% 5% t0% ts% 20% 2s% 30% 3s% 40% 4s%
Po rticipant Experience ond SotisfoAion
To assess customer satisfaction in the residential program and program elements, Cadmus included
questions around these topics in participant customer surveys. Overall, as in past evaluations, Cadmus
Exhibit 3
Case Nos. AVU-E-14 AVU-G-'|4
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 56 ol 127
40
observed generally very high customer satisfaction across the programs and program elements. The
sections below provide additional detail.
Overall Program Satisfaction
Cadmus asked surveyed participants to rate their overall satisfaction with the program as well as their
satisfaction with various program aspects. As Figure 16 shows, overall satisfaction with the programs in
PY2013 was very high, with 99% of participants describing themselves as somewhat satisfied or very
satisfied with the program in which they participated. This finding closely resembles findings from
PY2011 and PY2012, where 98% and 99% of respondents reported satisfaction, respectively. While
general satisfaction remained the same across program years, the proportion of participants that were
very satisfied rose steadily each year from PY2011 through PY2013.
Figure 16. Overall Participant Satisfaction across All Programs
89%
80% 83%
78% 160/^
t% O% t% l% L%I
Not at all satisfied Not very satisfied Somewhat satisfied Very satisfied
r 2011 (n=4611 .20L2 (n=645) 2013 (n=354)
As Figure 17 shows, participants expressed generally consistent, high overall satisfaction across
programs, with an appreciable increase in very satisfied Heating and Cooling Efficiency Program
participants from 2OL2 (82%) to 2OL3 (93%1.
4L
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 57 ot'127
Figure 17. Overall Participant Satisfaction by Program and Year
20L2 I 2013 I 2072 I 20L3 I 20L2 I 2013 I 20t2 I 20t3 I 20t2 I 2013 I 2012 I 2013
ENERGY STAR I Heating and I Space and I Water Heating lWeatherization I Appliance
Cooling I Water and Shell
Measures
Recycling
Efficiency I Conversions
I Somewhat satisfied r Very satisfied
Rebate Amount and Promptness Satisfaction
ln the survey, Cadmus asked participants how satisfied they were with the amount of the rebate they
received and how quickly they received the rebate.
Rebote Amount
As shown in Figure 18, respondents reported slightly lower satisfaction levels with rebate amounts than
with the overall program. This is not uncommon, as most peopled feel they would be made happier if
provided with a larger rebate. As shown in Figure 19, participants expressed generally consistent
satisfaction with rebate amounts across all programs. However, participant satisfaction (those who said
they were somewhat or very satisfied) with the Water Heating Program decreased slightly from 97% in
20L2to90%in 2013. lt is unclear what prompted this decline.
42
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 58ot 127
Figure 18. Weighted Rebate Amount Satisfaction for all Programs
Not At All Satisfied Not Very Satisfied Somewhat Satisfied Very Satisfied
I 2011 (n=4541 t2OL2 (n=632) 2013 (n=347)
Figure 19. Rebate Amount Satisfaction by Program and Year
2012 I 20t3 20t2 I 2013 20L2 I 20t3 20L2t20L3t20L2t2AL3 20t2 I 2013
ENERGY STAR
Products
Heating and
Cooling
Efficiency
and Shell
Measures
Appliance
Recycling
Space and
Water
Conversions
Water Heating I Weatherization
I Somewhat Satisfied r Very Satisfied
Promptness oI Rehote Poyment
As shown in Figure 20, respondents reported slightly lower satisfaction with rebate promptness than
overall program satisfaction, but slightly higher satisfaction than with the rebate amount. The
proportion of respondents who were very satisfied with rebate promptness increased slightly from 81%
in 2011 to 88%in 2OL2, but decreased to 80% in 2013. This may reflect the minor uptick in rebate
processing times identified in Table 16.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 59 of 127
Figure 20. Weighted Rebate Promptness Satisfaction for All Programs
LOO%
80%
60%
40%
20%
o%
Not at all satisfied Not very satisfied Somewhat satisfied Very satisfied
I 2011 (n=451) r 2012 (n=511) 2013 (n=340)
88%
8t%80%
t7%2OYo
o% olo 2% o%E
As Figure 21 shows, respondent satisfaction with rebate promptness was relatively high across
programs. However, the proportion of respondents who were very satisfied with the promptness of
their Energy Star product rebates decreased from 89%in 20L2lo 69% in 2013.
Figure 21. Rebate Promptness Satisfaction for All Programs
2072 I 20L3 I 2072 I 20L3 | 2Ot2 | 2013 I 20t2 I 2Ot3
ENERGY STAR I Heating and I Space and I Water HeatingProductslCoolinglWater and Shell
MeasuresEfficiency I Conversions
t Somewhat satisfied I Very satisfied
tM
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 60ot 127
Residential Progrom Freeridership and Spillover
Freeridership
Freeridership, the percentage of savings likely to have occurred in the program's absence, traditionally
refers to participants who would have undertaken an action promoted by a program had the incentive
or other program activities not been available. Full freeriders would have undertaken exactly the same
action at the same time (i.e., the program had no effect on the degree or timing of their actions). Partial
freeriders would have taken some action, but would not have undertaken the action to the level
promoted by the program, or would not have taken the action at the time they did.
For the PYzOLz - PY2013 evaluation, Cadmus estimated freeridership by measure type: appliances;
HVAC and water heating; and weatherization and shell using data from surveys with participating
customers. We established this grouping based on the needs of the impact evaluation. The customer
self-report approach to estimating freeridership adheres to standard industry methodologies. However,
the approach does present a potential shortcoming: it may not always be entirely appropriate for
capturing the market transformation impacts of multiyear programs. For example, a multiyear program
may alter the availability of higher-efficienry products in a region by influencing dealers' and retailers'
stocking practices. ln addition, by increasing dealer experience and comfort with more efficient
products, or by impacting demand for efficient products, a program may influence the mix of products
manufactured. Customers, when choosing between various makes and models of a given product, may
not be aware that a program affected their efficiency selection.
Therefore, while a customer may correctly state that he or she would have chosen a particular product
in the program's absence, the availability of that product may have been a result of the program. While
the customer would count as a freerider, the customer may have had less-efficient options without the
program. A more thorough description of the freeridership methodology is provided in: Avista 2012-
2013 Washington Electric lmpact Evaluation Report; and Avista 20L2-2OL3 ldaho Electric lmpact
Evaluation Report.12
Figure 22 show the freeridership results for the PY2O72 and PY2013 program, by fuel type. Estimates
from previous evaluations are also provided for context. Further, due to limited participants, before
PYlOL2, Cadmus did not break out freeridership scores by fuel. Cadmus did not calculate separate
freeridership estimates for conversion measures in PY2010 and PY2011 for the same reason.
" Avista 2012-2019 Washington Electric tmpoct Evaluotion Report. Cadmus.2oL4,
Avisto 2012-2073 ldoho Electric lmpoct Evoluotion Reporf. Cadmus. 2014.
45
Exhibit 3
Case Nos. AVU-E-'|4 AVU-G-I4
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 61 ot 127
Figure22. Observed Participating Customer Freeridership (Washington & tdaho)
79% 75jy.
NfrtIE
o(,
NdoN
Appliances
56% 55%
or
lc
o(,
(ndoN
6llc
G(,
moN
Nu,(cq
ooU(nHoN
F(o
ncodoN
hIc
.9
ooU
NHoN
HVAC & Water Heating
Froxc
ooN
tiorilc
dod
F
iil
I
oorNdoN
laUIdIc
oN
r\m
ilc
G(,
NoN
N@Ic
@(9
(ndoN
cottIc
.c
ooq
m
oN
F(ollE
odod
NmIc
oN
F6Iq
oIN
oN
Wx & Shell
uldIc
I'tru.EU
NdoN
Ic
.9
IoumdoN
A review of freeridership scores over the past four evaluation efforts indicates a clear upward trend in
self-report freeridership-particularly among appliance and HVAC measures. This finding suggests the
market for these equipment types may be transformed, and incentives from Avista are less of a factor in
customer decision-making. This supposition is supported by a review of available secondary data. As
indicated in Figure 23, which shows assumed appliance saturation in Washington and ldaho provided by
the NWPCC RegionalTechnical Forum13, there is little opportunity for customers to purchase and install
non-ENERGY STAR certified equipment. The NWPCC Regional Technical Forum estimates are derived
from the California Energy Commission (CEC) Appliance Database.
t'2014 NWPCC Regional Technical Forum Unit Energy Savings (UES) Measures and Supporting Documentation
http://rtf . nwcou nci l.org/measures/Defau lt.asp
Exhibit 3
Case Nos. AVU-E-14 AVU-G-I4
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 62oi 127
46
Figure 23. ENERGY STAR Appliance Saturation
LOO%
80%
60%
4OYo
20%
o%
Dishwasher Refrigerator
(2O7O-20L2 CEC Data) (2010-2013 CEC Data)
r ENERGY STAR Saturation
Clothes Washer
(2010-2013 CEC Data)
Freezer
(20to-2072 CEC Data)
Further, indicated in Figure 24 which shows average freeridership scores across all measures by
incentive amount (in 5100 bins), customers receiving smaller incentive payments are most likely to be
freeriders. As all Avista rebates for appliances were less than S50, it follows that freeridership is highest
in these measures.
Figure 24. Observed Participating Customer Freeridership by lncentive Amount
SOYo
I ToYoo
!eoNi soNoE toNooi 30%
o,f zoxoI tov,
o%
so-s1oo
(n=a1s)
s101-s200 s201-s300 s301-s400 s401-ss00 ss01-s600 s601-s700 s701-s800(n=69) (n=sa) (n=18s) (n=17) (n=9) (n=7) (n=s1)
IAverage Freeridership
-Linear(AverageFreeridership)
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 63 ot'127
Avista has already responded to high levels of observed freeridership in the appliance measure category
by discontinuing these measure offerings (Table 2).
Spillover
Spillover refers to additional savings generated by program participants due to their program
participation, but not captured by program records. Spillover also includes savings from actions non-
participating customers take because of program messaging or market effects. These savings are also
not captured in program tracking.
Energy-efficiency programs'spillover effects can be considered an additional impact that gets credited
to program results. ln contrast, freeriders' impacts reduce the net savings attributable to a program.
ln this evaluation, Cadmus measured spillover achieved through the installation of measures without
utility rebates through surveys with participant end-users and general population customer surveys
(representing nonparticipating customers). We found these savings to be the easiest to quantiry through
self-report suryeys, an approach in-line with evaluation best-practice.
ln these suryeys, we asked customers whether they had installed any other energy-efficient equipment
or had services performed in their homes for which they did not receive an incentive from Avista or
another organization. Next we cross-checked respondents against PY20L2 - PY2013 Avista and third-
party implementer databases to confirm that the customers had not received a utility incentive for the
reported measure. From this subset, Cadmus removed participants who did not indicate rebates or
information from Avista was "somewhat" or'tery important''to their decision(s) to purchase additional
measures and general population customers who did not indicate rebates or information from Avista
was "very important" to their decision(s) to purchase additional measures. Cadmus did not consider
appliances when calculating spillover savings due to saturation in the market of high-efficiency models
(Figure 23).
Table 23 summarizes the measures considered in PY2012 and PY2013 spillover estimates.
Table 23. Technologies Considered in Spillover Analysis and Number of Completed Surveys
Ctott "-r O.y"r
--
T.--
i Clothes Washer
48
Exhibit 3
Case Nos. AVU-E-14 AVU-G-I4
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page il ot 127
meosure
A!1s9aljng
Clothes Dryer
" _ F.l_"1{LqJrtlgqS
_ Fleclli! Watg! Hgatel
: Gas Furnace
r Gas Water Hggler _
' lnsulated Doors
lnsulation
2
1
Clothes Washer
Electric baseboard / Wall heatei !
1
8
3
5
3
62
1
- w;ath;iairippi.s
_Lrchlqc
Refrigerator
Wood/Pellet
Total
Windows
6
4
1
t2 42
27,5Survey respondents pet meosure 29.8
As indicated in Table 23, the number of spillover measures reported by respondents is consistent across
the various surveys fielded, with one measure reportedly being installed for 27.6 to 29.8 survey
respondents.
As a final step, Cadmus estimated energy savings from these additional measures installed, and matched
those savings to evaluated gross savings calculated for the sample of survey respondents. This led to
spillover ratios at the program levels. The spillover results for the PYzOLz and PY2013 are provided in
the Avista 2Ot2-20L3 Washington Electric lmpact Evaluation ReporU and Avista 2OL2-2OL3ldaho Electric
lmpact Evaluation Report.
Residentiol Conclusions ond Recommendotions
This section describes the evaluation's conclusions and recommendations for the residential programs.
Program Participation
Conclusion: Avista's implementation of new and continued support for existing third-party implemented
programs such as Simple Steps, Smart Savings and Residential Behavior effectively captures energy
savings in the residential market segments.
o Recommendotion: Continue exploring new measures, program designs, and delivery
mechanisms that leverage the national expertise of experienced third-party implementation
Exhibit 3
Case Nos. AVU-E-14 AVU-G-'|4
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 65ol 127
Participant(n=357) GeneralPoputation(n=1,109)
firms. Possible programs may include additional partnership with ENERGY STAR in the form of
the Home Performance with ENERGY STAR program.
Conclusion: Avista's continued investment in pilot programs provides a low-risk way test the
effectiveness of new measure offerings, delivery channels, and implementation partners.
o Recommendotion: Continue testing new program designs and measure offerings through the
use of pilots-even if secondary sources of funding or local partners are not available.
Conclusion: While still early, evaluation findings indicate the Residential Behavior program is an effective
way to capture savings in the residential market and Opower is a strong partner for program
implementation.
o Recommendation: lf determined to be cost-effective, consider expanding the Residential
Behavior program (for example, lowering the energy consumption threshold for participation)
and implementing measures to track the methods these customers use to save energy. Given
that Avista has already included all cost-effective customers in their target population for this
program, future opportunities for expansion may be limited.
Program Design
Conclusion: lnconsistencies continue to exist in measure and program naming and organization across
program planning, tracking and reporting activities which result in less transparency in program
operations and limit effective program evaluation.
o Recommendation: As part of the transition to the new data tracking system, consider aligning
program and measure names with offerings articulated in annual business plans and other
planning materials.
Conclusion: Reduction in Avista natural gas rebates and elimination of appliance rebates give customers
fewer ways to participate in Avista energy-efficiency rebate programs.
. Recommendation: Consider ways to encourage repeat participation (such as marketing targeted
at previous participants and online profiles that reduce application paperwork).
Conclusion: Considering self-report customer freeridership scores and market baseline data from the
RTF is an effective way to assess the appropriateness of measure offerings.
c Recommendation: Continue use of customer freeridership and market assessments as a way to
assess the appropriateness of measure offerings.
Conclusion: Many ongoing changes in Avista's program design and measure offerings are driven by the
need to continue to meet cost-effectiveness requirements. Avista's examination of measure and
program-level cost-effectiveness will determine the character of its portfolio in future program years.
50
Exhibit 3
Case Nos. AVU-E-14 AVU-G-'I4
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 66 ot 127
. Recommendotion: Develop a transparent process for assessing measure or program cost-
effectiveness and communicating results internally. Consider ways to ensure high-quality cost-
effectiveness analysis that aligns with industry best practices, such as obtaining an objective
third-pafi review of current cost-effectiveness screening processes.
Program I mplementation
Conclusion: Avista prioritization of customer satisfaction has been very successful and overall participant
experience is very positive across all rebate programs.
. Recommenddtion: Continue Avista's commitment to customer satisfaction, but monitor:
lncreased staffing costs; and
Impacts of the 90-day participation window on freeridership.
Marketing and Outreach
Conclusion: Avista implements a strong general awareness campaign around energy-efficiency, but
some room exists in market segmentation and targeting specific customer groups.
. Recommendotion: Utilize survey results from this evaluation and other data collection activities
to understand which audiences are more likely to participate in Avista programs.
51
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 67 ot 127
Nonresidential Process Report
lntroduction
This nonresidential process evaluation focuses on three Avista programs offered to ldaho and
Washington residential natural gas and electric customers during PY2012 and PY2013.14 ln this
evaluation, Cadmus sought to address the following researchable questions:
o What barriers exist to increased customer participation, and how effectively do the programs
address those barriers?
o How satisfied were customers with the programs?
o What changes to design and delivery would improve program performance?
ln assessing these topics, Cadmus relied on three main data-collection efforts:
o Review of program tracking data, documents, and invoice materials;
o lnterviews with Avista and implementation staff; and
o Telephone surveys with participating and nonparticipating customers.
Program Overview
Avista's nonresidential programs encourage commercial and industrial customers to install energy-
efficient equipment in their facilities. To accomplish this goal, Avista offers incentives directly to
customers who install qualifying equipment. This report provides findings and recommendations based
on a process evaluation of the three nonresidential energy-efficiency programs: Prescriptive; Site-
Specific; and EnergySmart Grocer.
Avista implements the Prescriptive and Site-Specific Programs. Avista account managers assist
customers and determine project eligibility for the Site-Specific Programs, while program engineers are
responsible for measuring and verifying project savings and costs. Trade allies also submit project
information and rebate applications on behalf of customers.
A third-party vendor, PECI, implements the EnergySmart Grocer Program. Energysmart Grocer is a
turnkey program available across the Northwestern United States.
The following sections provide descriptions of each program.
l4Similar to the residential portfolio, Avista's non-residential programs operate on calendar years, with program
years running from January through December.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-'|4
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 68 ot 127
52
Prescriptive Program
The Prescriptive program incents a variety of highly efficient electric and natural gas technologies,
including:
. PC network controls;
o Clothes washers;
o Food service equipment;
o Lighting;
o Motors;
o Variable frequency drives (VFDs);
o Windows and insulation;
o Heating, ventilation, and air-conditioning (HVAC) equipment; and
o Standby Generator Block Heaters.
Site-Specific Program
The Site-Specific Program offers incentives for energy-efficiency measures not included in the
Prescriptive Programs. All commercial, industrial, and water pumping customers with electric or retail
natural gas service from Avista are eligible for the Site-Specific Program. Site-specific measures consist
of electric and gas-saving technologies including:
o APPliances;
o HVAC equipment;
o lndustrialprocesses;
. Custom lighting,
o Motors, and
o Building shell improvements.
For a measure to be eligible under the Site-Specific Program, it must have demonstrable kWh or therm
savings.
The Site-Specific Program is responsible for a large portion of Avista's overall energy-efficiency portfolio
savings. This program generally offers an incentive for any energy-saving measure that has a payback of
more than one year and under eight years for lighting, and more than one year and under 13 years for
other measures. The incentive typically covers up to 50% of the incremental cost of the efficiency
investment.
Key drivers to delivering on program objectives include: direct incentives to customers, marketing
efforts, account executives relationships with large customers, and ongoing work with trade allies. The
Avista website is also used to communicate program requirements and incentives, and to provide
53
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 69 ot 127
application materials. The Every Little Bit and Efficiency Motters marketing and outreach campaign
(described in the Residential Process Report above) also focuses on commercial customers and is
designed to increase awareness of energy efficiency among commercial and industrial customers.
Ene rgySmo rt G roce r Prog ra m
The EnergySmart Grocer Program is a regional program that offers prescriptive rebates for a variety of
energy-saving food-sales and refrigeration equipment for nonresidential electric and gas customers,
with an emphasis on grocery stores. Eligible equipment incentives include:
o Compressors;
o Controls;
. Motors;
o Night covers for refrigerated cases;
o Case lighting;
. Strip curtains for refrigerated spaces;
o lnsulation for suction lines; and
r Hot water tanks.
This program helps customers with refrigeration loads to upgrade equipment, streamline operations,
and save energy. Customers receive a complete energy analysis of their facility's refrigeration and
lighting as well as a detailed report showing ways to reduce energy use. The customized report outlines
potential energy savings, incentive amounts, retrofit costs, and simple paybacks, and is offered at no
cost to the customer.
EnergySmart Grocer Program ollersTT prescriptive measures. The average program incentive covers
45% of the customer incremental cost of the efficiency investment-although in some cases the
program incentive covers up to 100% of the measure cost. Similar to the Site-Specific Program, key
drivers to delivering on the objectives of the program include: direct incentives to customers, marketing
efforts, account executives relationships with large customers, and ongoing work with trade allies.
Avista website is also used to communicate program requirements and incentives, and to provide
application materials
Evaluation Methodology and !nformation Sources
Cadmus' approach to this non-residential portfolio-wide process evaluation relied on four main reviews
and data-collection efforts. These activities and the program years they focused on are provided in Table
24. We applied activities to all three non-residential programs.
Table 24. Data Collection Activities Applied to Each Program
54
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 70 ol 127
Participating Customer surveys
N on_pa rtici patinBllqsto mer Su rveys
Rgalila!!on Rate and Databa_seBeview
Materidls Review
This process evaluation analyzes primary and secondary program data. Cadmus conducted the following
primary data-collection activities:
. Program staff interviews;
o Program participantls surveys;
. Nonparticipant customer" surveys;
o Database review; and
o lnterviews with lighting contractors.
Secondary data included the following program and marketing materials:
Avista's PYZOLZ and PY2013 DSM Business Plans;
An internal Avista program implementation manual;
Avista marketing collateral;
Everylittlebit.com website; and
Avistautilities.com website.
lnformation from Avista's reports for internal and external stakeholders, documents of public record,
and information about best practices also informed this evaluation.
Progrom Stoff and Market Actor lnterviews
lnterviews with program staff provided first-hand insights into program design and delivery processes,
and helped evaluation staff interpret the information collected. We conducted interviews with Avista's
Washington and ldaho program staff in two rounds, one in January 2013 and another in December and
January 2014.
Cadmus also conducted interviews with participating and nonparticipating lighting contractors in the
Avista service territory. These interviews were conducted in late 2013 as part of an ongoing Panel Study
Cadmus is conducting for Avista. The interviews included several questions designed to provide
feedback on Avista's programs from the perspective of participant and nonparticipant market actors.
Cadmus defined participating contractors as those with over 10% of their customers receiving Avista
incentives. Cadmus reached out to contractors on a list of 275 contacts provided by Avista, and offered
" customers who received a program rebate in 2OL2 or 20t3.
" Eligible nonresidential customers that did not participate in the programs during 2012 or 2013
,
a
a
a
a
a
Exhibit 3
Case Nos. AVU-E-14 AVU-G-I4
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 71 ot 127
an incentive for participating in the study. Of the 275 contacts, 157 were ineligible for the study either
because they were not commercial lighting contractors or because they operated outside of Avista's
service territory. Cadmus completed interviews with 20 of the remaining 108 contacts.
Table 25 provides a summary of interview data collection.
I Avista Policy, Planning and Analysis Staff i 1* l 2l
1*; Avista Marketing Staff
I ugnting contractors
I
--f n tuiirip;rl.]l
Ll(nonparticipont) ,;tvtrttipl. non-c.ar*rr statr partic-''p.t"a
''r'r
irrteri.;. -
Participant Surueys
Telephone surveys constituted a large part of PY2013 evaluation data collection activities. We
conducted all surveys with the assistance of several subcontracted market research firms, selected for
their experience with the commercial market segment. To minimize the burden on customers, ensure a
more satisfactory experience, and ensure high response rates, Cadmus designed the survey to take
approximately 15 minutes to complete.
The primary research objectives for participant surveys were to:
o Determine participant satisfaction with key program components and delivery;
Understand participant decision-making influences;
ldentify:
o lnformation sources and channels' effectiveness for outreach;
o Participants' perceptions of market barriers;
o Participant freeridership and spillover;
o Potential areas for program improvements and future offerings; and
o Compiling profile information about Avista's C&l target markets.
The process evaluation team used a single survey instrument for participants in all three programs,
maximizing survey efficiency by combining process- and impact-related questions into a single survey.
Cadmus designed participant survey samples to represent the programs proportionately according to
reported kWh savings. We adjusted survey targets to account for the number of survey respondents
available for a given program.
a
a
Exhibit 3
Case Nos. AVU-E-l4 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 72 ot '127
Table 25. PYlOL2 - 2013 Program Staff and Market Actor lnterviews
56
Table 26. Participant Survey Summary Details
-Prescriptive ..- . --- . --
-
79
j ste Specific i qt '.
Energy Smart Grocer
ldaho
i Prescriptive 33;
Site Specific 23
Energy Smart Grocer 11,_r3r * _i___ __ .t_:
Surveys were not conducted with PY2012 program participants because after conducting a large number
of surveys with nonresidential customers in 2010 and 2011, Cadmus and Avista elected not to conduct
surveys in 2072 to avoid survey fatigue in this population.
Non pa rti ci po nt Su rveys
The primary research objectives for nonparticipant surveys were to:
Determine program awareness levels and information sources;
Understand decision-making influences regarding energy-using equipment;
o lnformation sources and channels' effectiveness for outreach;
o Participation barriers or reasons customers aware of programs did not participate;
o Nonparticipantspillover;
o Potential areas for program improvements and future offerings; and
o Compiling profile information about Avista's C&l target markets.
2077-2072 Dotabase and Realizotion Rate Review
As part of the PY2012 process evaluation, Cadmus reviewed Avista's PY2OL? nonresidential project
database and project-level realization rates identified in Cadmus' PY2011 and PY2012 impact evaluation.
The materials reviewed and our associated research questions are listed in Table 27.
Table 27. Database and Realization Rate Review Activities
Database Review PY2012 SalesLogix Are data beingtracked accurately and consistently?
Database Extract , Are contracts issued in accordance with Avista policy?
74
a
a
a
57
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 73of 127
Do inientives comply with tariff rules for Washington and ldaho?
Realization Rate
Review
PY2011 - PY20L2
lmpact Evaluation
Sample
Why do rgr: proj".B h.*. r"ry lo* g!t1,:49l r.IL
Are there opportunities for Avista to improve the process of
calculating reported savings to improve the realization rates?
Database Review
Avista's tariff Schedules 90 and 190 govern how Avista can spend funds from the Energy Efficiency Rider
Adjustment paid by Washington and ldaho ratepayers.lT To assess compliance with these Tariff
Schedules, we examined two main indicators:
t. Project incentive amount: electric and natural gas project incentives should not exceed 5Oo/o of
the incremental cost ofthe project (p. 3 ofSchedule 90; p. 2 ofSchedule 190).
2. Project simple payback:
a. For lighting measures, the simple payback period must be a minimum of one year and
should not exceed eight years. (p. 2 of Schedule 90); and
b. For non-lighting electric and natural gas measures, the simple payback period must be a
minimum of one year and should not exceed 13 years. (p. 2 of Schedule 90; p. 2 of Schedule
1e0).
The tariff rules make exceptions for the following programs or projects (p. 3 of Schedule 90; p. 2 of
Schedule 190):
DSM programs delivered by community action agencies contracted by Avista to serve limited
income or vulnerable customer segments, including agency administrative fees and health and
human safety measures;
Low-cost electric/natural gas efficiency measures with demonstrable energy savings (e.g.,
compact fluorescent lamps); and
Programs or services supporting or enhancing local, regional, or national electric/natural gas
efficiency market transformation efforts. (ln 2012, Avista considered new construction fuel
conversions in multifamily building projects and T12 to T8 commercial lighting conversion
projects as market transformation efforts.)
Schedule 90: Electric Energy Efficiency Programs, Washington. Available at:
http://www.avistautilities.com/services/enerevoricins/walelect/Documents/WA 090.pdf; Schedule 190:
Natural Gas Energy Efficiency Programs, Washington. Available at:
http://www.avistautilities.com/services/enerevpricins/walgas/Documents/WA 190.pdf; and Schedule 90:
Electric Energy Efficiency Programs, ldaho. Available at:
http://www.avistautilities.com/services/enersvpricins/idlelect/Documents/lD 090.pdf
t?
Exhibit 3
Case Nos. AW-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 74 ot 127
58
Status of Evaluation Recommendations
Avista retained Cadmus to perform annual process and impact evaluations of Avista's non-residential
program portfolio beginning in PY2010. These evaluation activities, findings, conclusions, and
recommendations are articulated in the following reports: Avista 2010 Multi-Sector Process Evaluation
Report; and Avista 2011 Multi-Sector Process Evaluation Report.18
ln this evaluation effort, Cadmus reviewed the recommendations offered in these documents and
assessed to what degree Avista had adopted these recommendations (bythe end of PY2013). As
indicated in Table 28, Avista has made significant progress toward addressing these recommendations.
8
7t
Table 28. Status of PY2010 and PY2011 Nonresidential Process Recommendations
Complete
tq loqes1
Limited Activity
A complete summary of recommendations and activity for addressing these recommendations is
provided in Appendix B: Status of PY2010 and PY2011 Nonresidential Evaluation Recommendations.
Progrom Participotion
Savings and lncentives
Table 29 provides the number of incentive-based measures and reported savings. The PY2012 and
PY2013 Avista lmpact Evaluation Reports explore the reported savings in detail.
3,363 1,813 56,884 2L2,525
-,_39.,0_59
,, 504,577
Energy Smart Grocer 329 10,858
Total 4,3L7 2,470 106,792 717,096
'" Avista 2O7O Multi-Sector Process Evoluotion Report. Cadmus. 2011.
Avisto 2077 Multi-Sector Process Evoluotion Reporf. Cadmus.2072.
Table 29. PY2OL? - PY2013 Program Populations and Reported Savingsl
328
Exhibit 3
Case Nos. AVU-E-14 AVU-G-I4
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 75 ol 127
Progrom Design, Monogement, ond lmplementotion
This section discusses the Cadmus' observations regarding design and management of Avista's
nonresidential programs. These observations focused on program definition and organization, logic, and
implementation approach.
Overview
Overall, we found Avista's the non-residential program designs work well and are generally well-
documented, primarily in the PY2012 and PY2013 DSM Business Plans. Further, we found that Avista has
taken actions to improve internal communications and review processes.
Program Logic
Camus developed the logic model provided to articulate the logic behind the nonresidential program.
The nonresidential program's logic has not changed substantially since the previous process evaluation.
60
Exhibit 3
Case Nos. AVU-E-l4 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 76 ot 127
(ot C)N
=Ys(\l-O rir.=Y OLt+ i e,=EI
r(roD
4E8I(!d))o o<ofgFEz di5o'ds)8*o_EY
@
fr
ifiir
iuit
ft iff
i'Ii[
itriil-)i*il*.>
tr
IE
bt
}I
i$ !r -- i[*r -* il
il -'iii i!!i- ir fifr
E= ir --fl'iir - !iilnFir-iitiil-iliifr
!r-*lt*ifg,fr
il -'$fi lili* iil, - >
f r
E_tET.IT
ITc,t,T
&
&fiE
EFi!
TEE'
o'oo
=I'E!
oJ
El!
b!o
EL
E
0)
=o
ozt!.;
U;N
@
.!pr
ii{--
lnternal Communication
Avista's management and implementation of DSM programs has had some persistent organizational
challenges. While not limited to any specific part of Avista's DSM staff, many of the issues noted here
and in previous studies have primarily affected the nonresidential program internal review processes.
Several external documents and processes have addressed these problems, including:
2008 Ecotope lmpact Evaluation - cited potential for improved quality control
2009-2010 Moss Adams Process Evaluation Report - expressed need for central management
role and AA/aC checks in the nonresidential program
21t0-201t Cadmus Process Evaluation Report - recommended aA/aC checks at certain
threshold
August 2013 Cadmus Memo (see Appendix C) - review of 2072 program data noted some lack of
documentation, possible issue with application of tariff rules regarding payback periods and
incentive payment caps, and large variations between project-level realization rates
December 2013-January 2014 Cadmus interviews with Avista - noted internal disagreement
regarding whether the Top Sheet process was working
March 2014 ldaho Public Utilities Commission staff comments on Avista Corporation's
Application for a Finding that it Prudently lncurred its 2010-2012 Electric and Natural Gas Energy
Efficiency Expenditures - noted program implementation issues including a "lack of formal
follow-through on program management issues," "insufficient controls around engineering
assumptions and the basis for site-specific incentive payments, [and] incorrect interpretation of
Schedule 90 regarding implementation of prescriptive projects"
April 2014 ldaho Public Utilities Commission Order Number 33009 on Avista Corporation's
Application for a Finding that it Prudently lncurred its 2010-2012 Electric and Natural Gas Energy
Efficiency Expenditures - approved expenditures as prudent with the exception of incentives for
two projects for which recovery was deferred due to incomplete documentation, reiterated
need for a central decision maker
These documents focused on a variety of issues, but all documents agreed that there were concerns
with Avista's internal OA/aC process, especially for large nonresidential projects. These efforts agreed
that the definition of roles and responsibilities for Avista's DSM staff were not sufficiently clear. Further,
several documents noted that Avista's DSM staff was split into two completely separate teams: the
implementation team and the PPA team reported to separate directors. This separation may have
fueled internal communication problems.
Avista has taken significant steps internally to address these issues:
2009 Avista lnternal Audit Department review of DSM processes
2013 Avista retained Milepost Consulting for review of DSM team's roles and responsibilities
2013 Avista's implementation of Top Sheets - instituted peer review OA/aC system; associated
internal follow-up was completed to verify Top Sheet standard processes
a
a
a
a
a
62
Exhibit 3
Case Nos. AVU-E-I4 AVU-G-14
Khawaja, The Cadmus Group, lnc
Schedule 3, Page 78of 127
. July 2013 Avista lnternal Audit Department memo - noted that previously identified issues need
further attention
o April 2014 lnternal Audit Department memo - found that 70 out of 75 Top Sheets were present
and on-site verification is happening for tOO% of site-specific projects completed to date in
2014, but noted there is no policy on how many prescriptive projects should get on-site
verification
As of April 2014, Avista has begun a restructuring process to improve internal communication and
delivery of DSM programs. Both the implementation team and the PPA team now report to the same
Senior Director.
Effective ne ss oI I m ple m e nte rs
As noted in the Residential Process Report, using third-party implementers presents advantages and
disadvantages. Generally, utilities maintain direct implementation of programs requiring strong
relationships with unique customers (e.g., large commercial and industrial customers). Programs
benefitting from a uniform approach involve national accounts, or require certain market expertise
available from a third-party firm. Research conducted for this-and previous-Avista evaluation efforts
leads us to conclude that Avista has succeeded in identifying which program (EnergySmart Grocer) is
most suitable for third-party partnering.
The PY2011 evaluation report provides the results of detail interviews conducted with implementation
staff at PECI staff. As few changes have been made to this program since the interviews took place in
spring 2012, and the program has been the subjed ofother recent regional Cadmus evaluations,le we
did not conduct additional evaluation in this area.
Data Trocking, Verificotion, ond Quolity Assurance
Cadmus reviewed the PY2012 program tracking database for data accuracy and completeness, and
issued a memo in August 2013 describing in detail the methods, findings, and conclusions (Appendix C:
2012 Nonresidential Process Evaluation Memorandum). ln summary, we found some documentation
was lacking and that there were issues with the application of tariff rules regarding project costs and
energy savings specific to prescriptive projects.
We also examined the accuracy of Avista's claimed savings, measured by realization rates, and found
that accuracy improved significantly from 2011 to 2O!2. Three of the four main reasons for savings
adjustments in 2Ot2 were largely outside Avista's control. However, based on the review of 2Ot2 data,
" Cadmus recently completed an impact assessment and a market potential assessment of the EnergySmart
Grocer program in 2013. The results ofthis work are documented in reports available here:
http://www.bpa.sovlenersy/n/reports/evaluation/commercial/pdf/Cadmus ESG lmpact Evaluation Report Fina
l.pdf
http://www.bpa.sov/enerev/n/reports/evaluation/commercial/pdf/BPA Grocerv Opp Assessment 05JUN13.pdf
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 79 ol 127
63
we concluded that Avista could still improve the reliability of claimed savings estimates by avoiding
calculation errors in reported savings.
Cadmus reviewed achieved realization rates in each year, as summarized in Figure 25. This review
showed that the accuracy of claimed savings declined slightly in 2013, with 52% of electric project
realization rates falling within the 90o/o to LL0% range. This range reflects a high degree of accuracy, with
realization rate adjustments of 10% or less. lt is expected that some portion of projects will fall outside
of this range due to factors beyond Avista's control. Though the proportion of projects with realization
rates that fall below 90% is greater than that above 110%, the magnitude of those projects has been
steadily decreasing over the years, falling from 42Yo in 2011 to 29% in 2013.
Figure 26. Summary of Avista Nonresidential Project Electric Realization Rates
016 t@6 2@5 3@6 4@6
IRR=O% rRRBelow9o%
5@6 6@6 7cflo 8o95 gffio
r RR = 90 to 110% I RR Above 11096
ln July 2013, Avista instituted a new process for site-specific project reviews. A major feature of the new
review process was the addition of Top Sheets to track and verify applications' completeness and
correctness. Cadmus did not perform a review of the information contained within Top Sheets as part of
this process evaluation, but rather gathered information about the Top Sheet process through
interviews with staff.
Pa rtici po nt Ch a ro cte ri sti cs, Expe ri e n ce a nd Soti sfo dio n
To assess customer satisfaction with Avista's nonresidential programs, Cadmus included questions
around these topics in participant customer surveys. Overall, as in past evaluations, Cadmus observed
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 80 ot 127
64
very high customer satisfaction across the programs and program elements. The sections below provide
additional detail.
Participa nt Characteristics
Cadmus surveyed a total of 210 participating and 140 nonparticipating nonresidential customers. These
respondents represented a variety of business sectors, as shown in Table 30.
22%27%\6Yo 20%
17%20%7%6%
7%
74%
73Yo
6%
7t%
TLYo
9%
4%
6%
4%
70%
5%
3%
L7%
6%Warehouse / distribution center
Religious L%
Table 30. Participant and Nonparticipant Survey Respondents' lndustries, By State
Government building 9/o
3%
7%t%
6%
t%
3%
4%
oY:
4/o
Medical
ea*.tio" ix-rzt
6%
7%
Restaurant 4Yo 9%7%
Hospitality
Dormitory / multifamily housing
Education (college / university)
o%
L%
3%
3%
LYo
4Yo
3%
o%
1%3Yo
Agricultural 3%o%
Other lLYo
Program participant respondents were more likely than nonparticipant respondents to own their
facilities. lndicated in Figure 27,78% of participants owned their facilities, compared with 57% ol
nonparticipants.
LO%L6%74%
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 81 ot 127
65
Figure 27 . Facility Ownership Status, Participants vs. Nonparticipants
r Lease / Other
I Own
Participants (n=206)Non-Participants (n=135)
Most survey respondents, both participants and nonparticipants, used gas heating. Figure 28 shows fuel
use for space heating by customertype.
Figure 28. Fuel Use for Space Heating, Participants vs. Nonparticipants
60%
so%
4OYo
30%
20%
LO%
o%
Natural Gas Electricity
I Participant (n=206)
Both
(Gas & Electric)
I Non-Participant (n=132)
Participant Satisfaction
Overall, participants reported high satisfaction with the programsl. 84% ol all respondents said they
were'tery satisfied" in the program overall. Figure 29 shows respondents'satisfaction ratings by
program. ln contrast to the 2011 survey, when EnergySmart Grocer participants were less satisfied than
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 82 of 127
66
other participants, Energysmart Grocer participants reported the highest satisfaction levels in the
PY2013 survey.
Figure 29. Overall Participant Satisfaction
LOO%
80%
60%
40%
2OYo
Oo/o
Not too satisfied Somewhat satisfied Very satisfied
I Prescriptive r Site-Specific Energysmart Grocer
Satisfaction levels were generally similar across programs, as Figure 30 shows. However, the Washington
Site-Specific Program received slightly lower ratings than the other programs.
Figure 30. Participant Satisfaction, by Program
too%
8Oo/o
60%
40%
20%
O%o
Site-SpecificlPrescriptivelEnergySmartlSite-SpecificlPrescriptivelEnergySmart
GrocerlllGrocer
Washington
I Very Satisfied r Somewhat Satisfied
67
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 83ot 127
When asked how Avista could improve the program participation experience, Washington Site-Specific
participants suggested increased responsiveness and improved program information. Responses
included:
"lt would be nice if they could have recommend known heating and lighting and steered us to
the best installers."
"Contact me the first time I call."
"Find a way to do this sooner for better information."
"Just shorten the timeframe on the initial inquiry."
"lmprove the responsiveness of the technical team."
"Send me information that I need to finish the rebate process."
Participants also reported generally high satisfaction with individual program elements. As Figure 31
shows, at least 53% of survey respondents indicated they were "very satisfied" with each program
element. Avista staff received the highest satisfaction ratings, with92% of respondents "very satisfied."
Program materials were the element that received the lowest satisfaction rating, with 63Yo of
respondents "very satisfied." Participant satisfaction with the facility audit improved markedly since the
2011 survey, rising from approximate 509/o "very satisfied" in 2011 to 80% "very satisfied" in 2OL2-2OL3.
Figure 31. Percent of All Participants "Very Satisfied" with Program Elements
Program Overall
Program Materials
Rebate Amount
Application Process
Time-to-Receive Check
Facility Assessment
Quality of Contractor Service
Performance Of Measure
Avista Staff
Program Barriers
Participants reported facing several barriers to installing energy-efficient equipment. The most common
barriers cited are shown in Figure 32. The high up-front cost of energy-efficient equipment was the most
commonly cited obstacle; 50% of participants said it was a challenge. Next, 5% of participants reported
operational concerns, such as the inconvenience of having to work around customers and employees
a
a
a
a
a
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 84ot 127
68
during business hours, and a new oven that made the surrounding space too hot. Long return on
investment, lack of technical knowledge, and lack of staff time were obstacles according to 4% of
respondents. An additional 4% said there were no obstacles at all. A small group of participants (five
participants, or 2%ol had difficulty finding competent and trustworthy contractors and vendors. One said,
"The vendors twist information for their own benefit. lf they have different lights, they say [energy-
efficient lights arel not going to fit in there, so they install what they want to install."
Figure 32. Obstacles to lnstalling Energy-Efficient Equipment
High First / Upfront Cost
Lack of lnternal Resources
Operational Concerns
Long Payback Period
Contractor Concerns
o%LO% 2Oo/o 3oo/o
r 2011 r 2013
70%
62%
50%
Isr
liY;"
lzx
aqx
lll;N
to%
Program Benefits
Two-thirds (57%) of participants said the energy-efficient measures they took resulted in benefits
beyond energy savings. As Figure 33 shows, the most common non-energy benefit participants cited was
better equipment performance, such as improved comfort, better lighting quality, and less noise.
Additionally, 20% of respondents said the project increased productivity (including increased sales, for
retail facilities), while L2Yo cited lower maintenance costs. Other benefits that respondents mentioned
were less waste, environmental benefits, increased technical knowledge, and water savings.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-l4
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 85 ot 127
Figure 33. Non-Energy Benefits of Participation
I mproved equiprne nt performance
I ncreased productivity
Lower maintenane costs
Less waste
Environmental benefits
lncreased technical knowledge
Water savings
Morket Feedbock
Cadmus interviewed 20 commercial lighting contractors to obtain feedback on how Avista's programs
affected the overall market for energy-efficient lighting. Significant findings from these interviews are
provided below.
Contractor Awareness
The most common way the lighting contractors said they had heard about Avista's energy-efficienry
programs was through an Avista mailing. Figure 34 shows the sources of awareness the trade allies
reported.
Figure 34. How lighting Contractors Heard About the Programs
Avista Mailing
Past Experience with Programs
Avista Website
Avista Trade Ally Event
Supplier
Avista E-Mail
70
Exhibit 3
Case Nos. AVU-E-14 AVU-G-I4
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 86 ol 127
Program lmpact on Sales
Cadmus asked the lighting contractors what impact Avista's rebate programs had on their business. As
Figure 35 shows, 15 of the 20 contractors said their sales had increased, while four said they had seen
no effect. (None ofthe contractors said their sales had decreased due to the programs.) Two contractors
said they had noticed large increases in previous years, but that sales had dropped in 2013. One said,
"[the programs] increased sales when the T12-to-T8 rebate existed, but now it has no effect on sales."
Figure 35. Avista Programs' lmpact on Lighting Contractors'Sales
o
l!
to(,
o
olt
E5z
t2
10
8
6
4
2
0
I Non-Participant
r Participant
lncreased Sales No Effect on Sales
Nearly all contractors said energy-efficient sales would decrease if Avista's rebates were eliminated, as
shown in Figure 35.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 87 ot 127
71
Figure 36. Hypothetical Effect of Avista Rebate Elimination on Contractors' Sales
8
7
e6oulEE
toa4
o
O2!J
E:,=2
1
0
I Non-Participant
I Participant
Large Decrease Small Decrease
Market Transformation
Most contractors reported Avista's programs do not affect their stocking practices, as shown in Figure
37.
Figure 37. Avista Programs' Effect on Contractor Stocking Practices
o
.E
co(,
o
o!
E
=z
t2
10
8
6
4
2
0
I Non-Participant
I Participant
slight lncrease
72
Exhibit 3
Case Nos. AVU-E-14 AVU-G-'|4
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 88 ot 127
Morketing ond Outreoch
Program Marketing Approach
M a rketi ng Obj e ctives a nd Strategi es
Avista's marketing approach for 2OL3 was to increase awareness and participation in Avista's energy
efficienry programs for commercial and industrial customers using customer endorsements, and
showcasing additional value through non-energy benefits.
Planning ond Processes
Avista staff plan, design, and execute nonresidential program marketing initiatives. As indicated in the
PY2OL2 and PY2013 DSM plans, an internal collaborative process exists to develop general energy-
efficienry marketing and promotions. This process incorporates feedback from the Energy Solutions,
Services Development and Marketing, and Programs, Planning, and Analysis teams. The EnergySmart
Grocer Program includes supplemental marketing as part of its program design and implementation
plan.
Avista's marketing staff use the Avista Design System Guidelines to ensure that energy-efficiency
marketing and outreach materials deliver a consistent look, feel, and message. This document includes
guidelines for usages of items such as logos, color palettes, and fonts. lt also includes an overview of
applications, with examples of properly branded materials and collateral. All PY2012 and PY2013 general
energy-efficiency marketing materials appear to be aligned with the guidelines. The Efficiency Matters
campaign and Online Energy Advisor tool present slightly varied creative assets, although generally
appear to follow the brand guidelines (i.e., fonts, logos, etc.).
Outreoch Channels
Avista conducts residential energy-efficiency marketing through a variety of channels. ln addition to the
general energy-efficiency marketing tactics outlined below, Avista also conducts broad-based awareness
efforts through its Efficiency Motters campaign, as described in the following section. Besides the
Efficiency Motters campaign (which is implemented in partnership with KREM 2, a CBS affiliates), there
are no mass media or cross-cutting promotional efforts, to avoid potential customer confusion across
state lines. Notable outreach tactics used in PY2012 and PY2013 include:
o Paid media: print advertisements in local and regional magazines and newspapers;
e Earned media: local public relations as available;
o Direct mail and bill inserts: general and (targeted) program-specific;
r Newsletters and e-mail blasts: general outreach;
e Website (avistautilities.com): case studies added in 2013; and
o Vendor outreach meetings: general overview about programs, application process, project
qualifications, and customer eligibility.
73
Exhibit 3
Case Nos. AVU-E-14 AVU-G-l4
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 89 ol 127
The programs used print advertising to highlight customer success stories with call to learn more
information at two specialized webpages:
avistautilities.com/bizrebates
avistautilities.com/casestudies
Figure 38: Example Case Study Print Advertisement
"Cutting our heating costs in half
was just the beginning."
The ads appeared in select local and regional print publications, as shown in Table 31, targeted to reach
key business decision makers. The ads ran from May through December 2013, and delivered over
1,041,000 gross impressions.
a
a
Table 31. Print Advertisement Publications
74
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 9O ol 127
- HVAC/R lnsider
- The News (HVAC)
- Today's Facility Manager
Moterials and Messaging
Cadmus reviewed Efficiency Motters campaign outreach materials and Avista's energy efficiency web
pages, and conducted a high-level review of the Online Energy Advisor materials as a point of reference.
The evaluation team found that there are varied creative assets and look and feel across channels and
platforms. While the general energy efficiency promotional materials present a look and feel consistent
with the brand guidelines, the Efficiency Matters campaign and Online Energy Advisor platforms
leverage additional assets. For example, the Efficiency Matters landing page (www.everylittlebit.com)
also includes assets from the Online Energy Advisor personas (with the "shield" creative) and creative
developed by a 3rd party implementer.
Marketing Execution and Meosurement
Avista tracks metrics for its individual campaigns and ties results back to awareness and website traffic.
ln PY2013, Avista staff reported tracking Efficiency Moffers campaign metrics (participants and traffic),
estimated impressions through paid media, and response to direct mail.
Customer Awareness
Most of the customers surveyed had not heard of Avista's nonresidential programs; 38% of
nonparticipants recalled having heard about the programs. As Figure 39 shows, nonparticipants'
awareness has remained relatively stable since 2010.
Figure 39: Nonparticipant Program Awareness
45%
40%
35%
30%
25%
2lo/o
LSYo
!O/o
504
o%
- Spokane Journal of Business
- North ldaho Business Journal
- Coeur d' Alene Press
- Spokesman Review
- The Wall Street Journal (zoned)
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 91 ot 127
As shown in Figure 40, nonparticipants who were not previously aware of Avista's nonresidential
programs overwhelmingly say they want to hear about them through the mail - bill inserts or direct
mail. Nearly a quarter reported wanting to hear about the programs through e-mail.
Figure 40. How Nonparticipants Want to Hear about the Programs
Notices in utility Bill
Direct Mail
E-Mail Updated from Avista
Avista Account Representative
Social Media
Other
Sources oI Porticipont Awdreness
ln both Washington and ldaho, most participating customers reported hearing about the program from
a contractor or vendor, as shown in Figure 41. Contact from Avista and word-of-mouth were also
commonly reported sources of awareness in both states.
Among Avista's marketing efforts, the program website was the most commonly cited source of
awareness, wilh 7%. Three percent each said they learned about the program from printed materials
(such as flyers or brochures) and the electronic newsletter. No participants reported they heard about
the program through magazine or newspaper advertisements.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 92 ot 127
76
Figure 41. How Respondents Heard About the Program (Participants - tdaho)20
Contractor/Vendor
Word of Mouth
Contacted by Avista
Customer Contacted Avista
Program Website
Rebate for Other Measure
Electronic Newsletter
Print Materials
Trade/Professional Organization
Program-sponsored Event
Other
Nonresidentiol Progrom Freeridership ond Spillover
Freeridership
Freeridership, the percentage of savings that are likely to have occurred in the program's absence,
traditionally refers to participants who would have undertaken an action promoted by a program had
the incentive or other program activities not been available. Full freeriders would have undertaken
exactly the same action at the same time (i.e., the program had no effect on the degree or timing of
their actions). Partial freeriders would have taken some action, but would not have undertaken the
action to the level promoted by the program, or would not have taken the action at the time they did.
Table 32 shows overall nonresidential freeridership results for 2013, including gas and electric projects
and participants in both Washington and ldaho. These results are based on 2013 participant survey
response data and weighted by project savings.
20 Percentages may add up to more than 100% because respondents were permitted to give multiple answers,
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 93ol 127
Table 32. Nonresidential Freeridership Estimates PY2013
The PY2013 prescriptive program showed a low level of freeridership, while the site-specific program
showed slightly over 30% freeridership. As shown in Figure 42, these results differ from 2011
freeridership results, but are fairly similar to the results found in 2010.
Figure 42. 20LO,20t1, and 2013 Nonresidential Program Freeridership
3s%
3@6
25%
zffi
L5%
L@6
5%
M
2010
.2011
r 2013
Prescriptive FR Energy 9nart Grocer
FR
Site Specific FR
Because nonresidential projects can be very large, and freeridership results are weighted by savings, the
highest saving projects in the sample can have a strong influence on year-to-year results. To further
examine the difference between the 2013 and 2011 analysis, Cadmus identified the top three savers in
each program category and their freeridership scores.
Prescriptive showed a decrease in freeriderchip: A key driver of the decrease is that in the 2011
analysis, the three respondents with the highest gross energy savings accounted for 34% of the
survey sample's total gross savings. The top energy saver was estimated as a75t%treerider, and
represented 19% of the total survey sample savings, while the second and third highest energy
savers were estimated as 0% freeriders. ln 2013, the three participants who achieved the
greatest savings accounted for 38% of the total gross savings for the survey sample and all three
respondents were estimated to have 0% freeridership. As such, the high level of savings
achieved by these three 2013 participants, relative to the rest of the 2013 survey sample,
resulted in these participants'freeridership scores greatly reducing the overall freeridership
estimate reported in 2013 compared to what was observed through the 2011 evaluation efforts.
Energry Smart Grocer showed an increase in freeriderchip: A key driver of increase is that in the
2012 analysis, the three respondents with the highest gross energy savings accounted for 72% ol
the survey sample's total gross savings and all three respondents were estimated to have 0%
freeridership. As such, the high level of savings achieved by these three participants, relative to
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 94 ot '127
78
the rest of the survey sample, resulted in these participants' freeridership scores greatly
reducing the overall freeridership estimate reported in 2011. ln 2013, the three participants
who achieved the greatest savings only accountedfor 64% of the total gross savings for the
survey sample and the top energy saver was estimated as a Oc% freerider, The second largest
energy saver, representing 16% of 2013 survey sample savings, was estimated as a 75%
freerider and the third highest energy saver as a 0% freerider. As such, the high level of savings
achieved by these three 2013 participants, relative to the rest of the survey sample, resulted in
these participants'freeridership scores greatly increasing the overall freeridership estimate
reported in 2013 compared to what was observed through the 2011 evaluation efforts.
. Sate-specific showed an increase in freeridership: A key driver of the increase is that in the 2011
analysis, the three respondents with the highest gross energy savings accounted for 35% of the
survey sample's total gross savings, and first and second highest energy savers were estimated
as 0% freeriders, and represented 28% of the total survey sample savings, while the third
highest energy saver (7% of total survey sample savings) was estimated as a 100% freerider. ln
2013, the three participants who achieved the greatest savings accounted for 4to/o ofthe total
gross savings for the survey sample. The top energy saver, representing2t% of the survey
sample savings, was estimated as a O'% freerider. The second highest energy saver was
estimated as a 5Oo/o freerider and the third largest saver as a tOO% freerider. As such, the high
level of savings achieved by these three participants, relative to the rest of the survey sample,
resulted in these participants' freeridership scores increasing the overall freeridership estimate
reported in 2013 compared to what was observed through the 2011 evaluation efforts.
These year to year variations accurately reflect the activity of participants within each program year, but
they can reduce clarity when observing year-to-year trends. For example, since the site-specific program
did not change substantially between 2011 and 2013, the large change in freeridership may reflect
differences between individual customers, rather than changes in the market or in the program's
implementation. Therefore, Cadmus also calculated combined freeridership values that reflect the
aggregated survey data from 2011 and 2013. These values may portray a more reasonable estimate of
the programs' overall level of freeridership that could be expected in future years if programs do not
change substantially.
Table 33. Nonresidential Freeridership Estimates: Combined PY2011 and PY2013
Prescriptive
Energy Smart GrocerL_,,_..
Site-Specific
i.t"l "
L9.5%
79
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 95ol 127
Spillover
Participant spillover refers to additional savings generated by program participants due to their program
participation, but not captured by program records. Spillover occurs when participants choose to
purchase energy-efficient measures or adopt energy-efficient practices due to a program, but choose
not to participate (or are otherwise unable to participate) in an incentive program. These customers'
savings are not automatically credited to the utility program. Energy-efficiency programs' spillover
effects can be considered an additional impact that gets credited to program results. ln contrast,
freeriders' impacts reduce the net savings attributable to a program.
ln this evaluation, Cadmus measured spillover achieved through the installation of measures without
utility rebates through surveys with participant end-users. We have found these savings to be the
easiest to quantify through self-report surveys.
As shown in Table 34, Cadmus found a small amount of participant spillover for PY2013, equivalent to
0.05% of total program gross savings. The reported measures included in the spillover savings included
LEDs (350 total units) and energy-efficient light fixtures (10 total units).
Prescriptive
Energy Smart Grocer
Site-specific
Total
204,728
0
14,L48,704
r+,iii,iigs
7,8L2,790,682
2,885,O93,927
ii,sze,g$,zqt
30,536,803,843 I
O.OOo/o
o.o0%
o.o7%
o.o5%
Nonresidentiol Conclusions ond Recommendotions
This section describes the evaluation's conclusions and recommendations for the nonresidential
programs.
Program Management and lmplementation
Conclusion: Several parties over several years, internal and external to Avista, have observed the need
for greater data quality assurance, in both documentation and input tracking. Quantitative inputs to the
savings and rebate calculations have repercussions for tariff compliance,zl incentive payments, and
savings rea lization rates.
o Recommendotion: Avista should continue efforts to improve program processes. Cadmus
understands that a reorganization ofthe DSM group has occurred concurrent to the delivery of
this report. This change may be an opportunity for fresh perspectives, clarified responsibilities,
21 As noted in ldaho Public Utilities Commission Order Number 33009 on Avista Corporation's Application for a
Finding that it Prudently lncurred its 2010-2012 Electric and Natural Gas Energy Efficiency Expenditures.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 96 ot 127
Table 34. Nonresidential Spillover Estimates for PY2013
80
and improved coordination within and between teams. We believe unifying the organizational
structure under central leadership is a step in the right direction and may help alleviate some
previously documented issues with internal communications.
ln addition to the reorganization, Cadmus recommends that Avista develop standardized
processes within the DSM group, including clear delineation of roles and precise description and
assignment of all processes and responsibilities for both residential and nonresidential
programs. All affected parties should be included in formalizing and standardizing the DSM
group's processes, roles, and responsibilities. Further, all parties must formally agree to clearly
delineated responsibilities under the new organizational structure. While these activities need
to be prescriptive and precise, we caution that the resulting structure should still allow some
flexibility: increased clarity, transparency, and accountability should serve to enhance program
delivery and customer satisfaction.
Customer Feedback
Conclusion: Customers were highly satisfied with the program overall and with individual components.
Customer satisfaction has increased since 2011, which had in turn increased from 2010.
r Recommendation: Continue to prioritize and monitor program satisfaction.
Conclusion: Customers appeared to be slightly less satisfied with the Washington Site-Specific program
than with other programs. The Iargest source of lower satisfaction was the participants' reactions to
program materials. Many customers said they received no program materials, and many participants
learned about the program from their trade allies.
o Recommendotion: Consider taking action to strengthen the use of program materials. Consider
providing trade allies with printed program information flyers or brochures to give to customers.
Maintaining up-to-date information for trade allies is critical when they are the key party
delivering the program's message and participation details.
Market Feedback
Conclusion: According to commercial lighting contractor feedback, the nonresidential programs are
successful in driving incremental energy-efficient equipment sales, and the market has not yet
transformed to make energy efficiency standard practice.
o Recommendotion: Continue to monitor market transformation indicators to measure programs'
market impact over time.
Marketing and Outreach
Conclusion: The characteristics of Cadmus' survey respondents indicate that the office / professional
services and local government sectors may be underserved by the programs relative to their incidence in
the nonparticipant population. Further research is necessary to determine whether this is true.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 97 ot 127
. Recommendation: ldentify underserved industries, and seek opportunities to target outreach to
specific underserved industries:
lnvestigate overall customer industry distribution
Compare to participant industry distribution
Develop targeted outreach strategies for any underserved sectors
Quality Assurance and Verification
Conclusion: Avista monitored its site-specific project review process and instituted refinements during
the evaluation period in response to feedback from users. While this has led to improvements, including
notably improved reliability of reported savings in2OL2, quality assurance problems may persist.
o Recommendotion: Continue to monitor the effectiveness of the site-specific project review
process and refine as needed. Cadmus recommends implementing the following to ensure
continued imProvement:
All large prescriptive or site-specific projects reporting savings over a threshold of 30O000
kWh or 10,000 therms should undergo a complete AA/QC review prior to incentive payment
in addition to the standard Top Sheet review process. Typically, a QA/QC process reviews
engineering calculations, verifies inputs, checks payback period and incentive payments for
reasonableness, and ensures compliance with program requirements and tariff rules. ln
order to align with the above recommendation regarding program management and
implementation, Cadmus recommends that Avista determine and document the specific
requirements and steps in the QA,/QC process through a collaborative process that will
ensure accountability and balance needs for efficiency and customer satisfaction.
Conduct an external third-party review of Top Sheets, including reviewing a random sample
of completed Top Sheets for completeness and accuracy. These were not reviewed as part
of the current process evaluation, but should be included in the next process evaluation.
Review should not only verify the presence of the Top Sheets, but also the quality and
accuracy of the information provided.
o
82
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 98 ot 127
Appendix Status of PY2010 and PY2011 Residential Evaluation
Recommendations
Table 35. lmplementation of PY2010 Residential Evaluation Recommendations
Program Participation
Research market saturation and participation to track achievement of potential.Complete
Using the Avisto Electric Conservotion Potentiol Assessment Study completed in August 2071, olong
with ovoiloble data sources such as ENERGY STAR ond odditional primory reseorch, Avisto should
track the residential portfolio's progress toward copturing projected realistic ochievoble potentiol.
This effort will inform progrom plonning ond design decisions to ollow for the long-term success of:,__ the residentigt gort!9tp.I Discontinue rebate for ENERGY STAR dishwashers.
ENERGY STAR dota shows thot 78 percent of dishwoshers sold notionolly are ENERGY STAR models.
Therefore, this meosure is likely to suffer from high freeridership, ond the Avista rebote is unlikely
to alfect morket tronsformotion.
Emphasize ease of participation in marketing.l! Lrggr"I_
ln order to address the nonporticipont perception thot progrom participotion may be difficult,
Avisto should empllsizlthg eote gl pjrylcipg_ti1tO in relideltlg!morkeling
Program Design
Simplify and document p.gr., organizaiion structure.
Codmus recommends grouping progroms in logical clusters, in order to reduce complexity of
documentotion ond tracking. While streomlining program organization, Avisto should also
ln Progress
document institutionol knowledge of progroms to ovoid loss of continuity,
Avisto should consider the suggestion from HVAC trode ollies to provide rebates direct to
contractors. Other utilities hove seen success with this model, which reduces the administrotive
burden on customers, ollows for botch processing of rebotes by Avisto, ond ensures close
communication with trade ollies, Anti-fraud provisions (such os requiring customer informotion and
signoture on rebote forms, or conducting site visits to veriJy installotion) must be included in ony
luch p rog ra ry gd_qptgylon.
Data Tracking
lglsidel ennqlging glllolmity oJ prqrygrnlacking by standa-{gfg_data torrygts.
Wherever possible, Avista shou!d develop trocking methods thot support consistent onalysis ocross
progroms. For example, a standordized format for customer oddress dota across separote
Complete
qqtubasgsyglggls:!otgP_?:e_99,n!!dt!9norllt:grgtlo_l:___
Track follow-through on audit recommen{4i_of :.ln Progress
ln plonning for Iuture Audit program implementotion, Avisto should consider odditionol trocking oJ
customer follow-through on recommendotions, both through othet Avisto rebote progroms, ond
targeting difficult-to-reoch customers where o ppropriote.
Continue enhancing social media marketing.Complete
83
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 99 ol 127
Since Avisto rcported thot younger customers con be more dilficult to reoch, the marketing teom
-___:ne:t!aSgti!1t19togn|a!!elt1sogioly-algmq!\fi 9_9ff 9Is.
Ensure contractors have adequate information to disseminate.Limited Activity
Since trode ollies were one of the commonly reported woys that porticipants leorned obout the
progrom, Avisto must locus on providing trode ollies with odequate ond occurote informotion. This
con be ochieved by distributing updoted moteriols regulorly, holding troinings for controctors, or
formolizing the trode olly network to ensure frequent communicotion. For exomple, Avisto should
consider providing printable online information sheets that trode ollies con print ond disseminote to
their customers.
Participant Experience and Satisfaction
Continue emphasizing good customer service and offering customer-friendly programs.lgqulqte- ,;
These oreas should be mointained os priorities in future progrom plonning and implementotion.
Effectiveness of !mplementers
_- ConsiderexpandinsoferyCryQirq!9!!9P[recElL__ _, _ qqqplelg_
Avisto should consider the benefits of adding measures to the Simple Steps progrom. Additionol
meosu re oIIe rinss moy'!cr9! selot!!!!g!yg9ag!!g! g!!:!y!gs.
Require [CLEAResult] to ensure evaluators have access to retailers.Limited Activity
Upstreom progrom evoluotion often requires occess to retail locations, for shelf-stocking studies
ond in-store intercepts, for exomple. ln order to ensure future evoluability of the Simple Steps
: program, [CLEAResult] should require potticipoting rctoile$ to gront such occess to evoluotors
I l! Prosrggl
Avista should olJer odditionol troining and informationol moteriols to contrcctors who serve the
HVAC progrom, to ensure high-quolity progrom information reoches customers, ond to encourage
progrom promotion through contractors.
Residential Portfolio
Consider various opportunities for expansion,1 cgmpl_elg , I
Avista should regulorly ossess the viobility of expanded progrom and measure offerings. Avisto moy
co n si de r vo ri ous possi bl e expa nsions i n cl ud i n g :
- Adding showerheads to Simple Steps
- Additional cost-elfective meosures in HVAC progrom
- Behaviorol programs, energy educotion progroms
Table 36. lmplementation of PY2010 Residential Evaluation Recommendations
Renew emphasis on customer outreach and mass marketing, including refreshing campaign
messaging and using trade allies.Complete I
Consider using lessons learned from the Home Energy Audit Pilot Program to design and implement a I Limited Activity l_trtt-r..t" p.S .t audits or a similar whole-house approach.
_Program Desig
Consider additional prog[nr r{;ir"r""tt t" ".",- measure savings remain in line with Limited Activityexpectations. _._
84
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 100 ot'127
For exomple, Avisto should revisit progrom eligibility for multiple meosures, where sovings are
interactive (porticularly for HVAC equipment), ond consider adjusting sovings to reflect interoctive
effects, or incenting specific pockoges of complementory measures. Avisto may olso consider not
_*9$-erinq hgoj pump incg\iv9s when notural gas is av9i.l9.b!e.
Explore possible benefits of outsourcing simple rebate processing for ENERGY STAR appliances and
hot water heaters in order to allow program managers to focus on long-term program
lons!derl!!911
Market Characteristics
Ensure future program effectiveness by continuing to update program offerings and design to reflect
44e"t In r,Jkgt c ndltiqE
Data Tracking
Ensure consistency in data tracked across multiple databases including: the multi-program database;
the JACO database; the Home Energy Audit database; and Avista's central customer information
database.
If Avista continues the Home Energy Audit Program, audit tracking should be enhanced to include:
integration into the central participant rebate database; and more robust tracking of data collected
through the audit, and of follow-through installations.
Marketing and Outreach
Avista should maintain its multifaceted approach to reaching a broad range of customers, while
tgrget4q dfficgL!-to-re?c]1c_1.rstomers, where appropriate. Pory]lle rygb1te enhanggm_e{sllg]rd",
- Exploring relotionships between the corporote website ond EveryLittleBit.com. Explore the
Entronce-, Exit- and ln- Poge onalytics to ochieve o deeper understanding of the poths people toke
within the website,
- Adding o content-sharing toolbor to the EveryLittleBit.com website to promote refenols. This
toolbar would ollow users to shore content vio email, RSS feeds, or sociol media plotforms.
Participant Experience and Saiisfaction
Continue to prioritize customer satisfaction, and take advantage of high satisfaction by targeting past
participants for future participation.
Residential Program Freeridership
Continue conducting research to inform decision making about future program
improvements/contin uation.
ln Progress
Complete
ln Progress
ln Progress
ln Progress
Complete
Complete
Effectiveness of !mplementers
Explore posslb!e benefits of third-party pfogram implementation.
Avisto's newly lounched online rebate applicotion system moy olleviate stoff burden associoted
with rebate processing. However, that tronslerring responsibility for rebote processing to o third-
porU contrdctor could convey further benefits. Specificolly, this option should be explored lor the
ENERGY STAR Applionce Rebote Progrom dnd woter heoters, os the applicotion reviews for these
measures do not require o high level of expertise.
ln Progress
Trade Ally Participation and Satisfaction
3ysta 9h.ollqjlyestlqqlglhe pgqibrlity 9l! ln9l9_r_-nflryl9l,olghirlqrlh !ra!9"el!St._ __11[ogry1s
This would ollow increosed progrom morketing through trode olly channels, while ensuring
occountobility ond professionolism. Disseminoting simple program informotion sheets to
controctors ond retailers would be o low-cost, first step toword developing relotionships with key
trode ollies. More involuement might include, for exomple, hosting trode-ally training events,
85
Exhibit 3
Case Nos. AVU-E-14 AVU-G-l4
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 101 ol 127
Appendix B: Status of PY2010 and PY2011 Nonresidential Evaluation
Recommendations
Table 37. lmplementation of PY2010 Nonresidential Evaluation Recommendations
Program Documentation
oevetopini a progr.r r.nr.L *ith i;Gdtation plani operationat pro."oriis, mirketing
strategies, and verification protocols aggregated into a single program handbook, could help to
establish a link between EM&V policies
p_ro_gF-m:s_o p,e_r?!!e tqi m ?I laqll 9L' !: -
Customer Feedback
found in the high level planning documents and the Complete
A!{qss gq!9ner+9f.!'r9q_E!k qt l41|of ,n+tsqeboylqrrgtgn ojq{!_s!.__i lnnonce outrerrn *a ,ii^"rirrt'rr;ffortt f", p"rti;prni, iorportiriponts, ond portiol
participonts.
o Develop odditional printed progrom moteriols to educote customers about program
opportunities.o Consider regularly scheduled online Webinars to assist customers with questions about
-_ _ .*_?Jggrom !!lcgn!!Y1 s!!g!9tti!y,S!! igJtSgllgn p!999iq,tqs.
r*Ingellly ?qrticlpdo,l eu!:tElr+igt,
-
Provide regular trade ally communications through targeted outreach efforts, such as a Website,
monthly e-mails, or a newdetter.
A Website dedicoted for trade ollies could enoble registrotion, thereby providing a method for
gompitilo gOq ugdotilol!!!!!S:!tv prol!192 o7! conto* !n[9rm9ti9n.
Consider providing additional promotional materials that would highlight various program
_t9_cht'_S!9sie4yq44]9 !9:9!!!!9rn9!s, rhi-s rry!fup! feq!.!I9_th!!4yLs!a ellqqq rny 9Le ce!tr1$9l,
Explore ways to leverage strong working relationships forged between customers and contractors
within the community by sponsoring additional program working sessionq luncheons, or Webinars
downloadoble ond moilin would provide o good first step. ln oddition, consider including guidelines
f91 co npQtn g site-specifi c form s-.
Gather additional feedback from customers and trade allies about how site-specific form enrollment
and processing could be_:treamlined.
Gathering more detail about program and project measures in the participant database would enable
a better understanding
ln Progress
Complete
Complete
Complete
Limited Activity
end-uses).
ln Progress
of the kinds of projects done in the past (by different types of customers and i ln Progress
!
Additionol informotion could be used to market specific types of projects to other customers who
hove the some end-use equipment.
_y alkglln g ary! 9u! rgfch
Ensure allocation in future marketing budgets dedicated for nonresidential program marketing and i: , , ,__*_ _ ,_ ::.|l'_'" ,l
trev('lop qqqltional_marketing m4g!:la_ls_t3lc_ejg!_specifically for tra49 4ly!!!Ig1ch!9.c1!!grngrs. - lolplgle _
These moterials would enable Avisto stalf to leveroge existing trode olly relotionships in the
- -cgryly91_mq1!91'nq.,rlrqy_!,
qplt{setgq qqlgtjng regg?Ig! th4w9!ld_q{!9l addLtio!.d l-!F[-ed l9!']qy_ l
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page lO2of 127
86
information about customer facilities and technology end-uses.
Conduct targeted marketing research of largest 100 customers with hourly demand data. Limited Activity
Use such dota to anolyze demond potterns, identify opportunities, and provide occount executives
w4! rye_ded ilteillsgy:lg marlgt eryrov 9fr9iency ry:9:yres.
Quality Assurance and Verification
Consider developing a verification protocol to document pre- and post-inspection procedures for
prescriptive programs, and ensure data tracking for project installation, ln addition, protocols should ln Progress
highlight any differences in verification procedures used for prescriptive and site-specific programs.
Table 38. lmplementation of PY2011 Nonresidential Evaluation Recommendations
Program Management and lmplementation
Consider a method for prioritizing management tasks, thus enabling allocation of more time for
nla11!1g and {evelopme11of program documentation.
Revisit the staffing needsfor delivering the current programs.
Revisit the opti-n of using third-party implementers for some programs.
Consider round tables with the program implementation, managemen! and policy team to facilitate
additional communication regarding planning and evaluation.
Consider designating a centrai leidership role for the Site-Specific Program to oversee future
planning and vision, and ensure that it continues to deliver cost-effective energy savings to the C&l
portfolio.
Further investigate contractor issues to ensure high satisfaction levels of Energysmart Grocer
program participants
program offerings to small-to-medium customers.
Further explorotions could determine if controctors offer better morket coverage, are more likely
to connect with customers when purchoses are being contemploted, provide o more compelling
value proposition, or offer other lessons Avisto could opply, both with contractors ond ocross
oth e r com m u n i cotio ns ch o n n el s.
Strategies should be developed to penetrate leased C&l spaces, targeting building owners, managers,
and brokers of leased space. Examples could include:
c Toilored messoges, delivered through presentations or workshops in conjunction with the
Building Owners and Managers Associotion ond commerciol reol estote associotions.
e Designoted point-of-contoct ond web informotion for building monogers ond brokers.
o lncentive ond ftnancing solutions, such os on-bill finoncing, green leose orrongements, and
6snu5 inrgntiues tgfgeting Let!9Ftts wh91 new tenonts move in.
Cadmus recommends Avista evaluate alternative strategies for reaching small-to-medium businesses
cost-effectively via contractors, direct install, or more Prescriptive, "self-serve" options via the Avista
website. Such strategies could include:
. Promote newsletter sign-ups and explorotion of progrom information on the website.o ln progrom informotion, cross-reference sources or the avoilability of onswer lines.
o Evoluote measures installed by smoll customers in the Site-Specific Program for inclusion in o
Prescriptive progrom.
ln Progress
ln Progress
Limited Activity
Complete
ln Progress
Complete
Complete
ln Progress
ln Progress
87
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 1O3ol 127
Where customers expressed lower satisfaction levels, program elements should be investigated.
Such investigatioLs misht include:
o Review oudit program communicotions ond supporting colloterol to improvle custoiiers'
understonding of the depth of oudits, ond recommendotions. Consider providing informotion
obout economic odvontages to energy elficiency such as improved benefi* to costs rotios,
ond simple poybock.
. Determine/trock cycle times for customer follow-up ofter audits ond for rebote opplicotions;
if reosonoble times are exceeded, consider implementing follow-up communicotions to keep
customers informed and ensure internol Jollow-up, if needed,
_ - ._ _99!ti!l!j::!9l ilgntJlE! ln !h_e-!y1gyfgg4 srysetp-rgslgll hgygbeen resotved
Trade Ally Fgqdb.ack
eipiore rnoie foirali.ed *;t;-tfioiraae itties in promoting noniesioentiar programs to customers.
Avista should continue efforts to expand outreach to trade allies, through sponsored events and
_ryg(slgqs, F19_?$aqlpeellngjrfqgylgt_o_gpj, qtrq_o!hellqlsgtg!_9o,mmu1qg!!9ts_: .._,_* . .
Given trade allies' requests for a dedicated Avista contact, more one-on-one communication, and
additional materials to inform customers about the programs, more timely feedback could be
achieved through online resources. These resources may also help to reinforce the program's
tr9:tsCqlg{9!nil9qofl99sl!tstls!_[!!!!PE Oa!!e]r !v ptoytdtnel!9_lo!!9yvlqg 9er"v_1qes.o Olfering o dedicoted website, contoining guidonce through webinors ond video presentations.
Online registrotion for events or informotion requests.
An online help desk or phone hotline, which would direct customers to onswers for frequently
asked questions, or would reseNe more complicated questions for progrom stoff.
Other, odditional promotionol moterials, posted online, such as hondouts regarding costs and
Take a more proactive role in communicating with customers:
ln Progress
Complete
Complete
a
a
Complete
o Upcoming changes in lighting product ovoilobilityc Avisto's program ovoilobility to olferthem helpo When the T-72 proqrom will end
o Communicotions should also olfer help in identifying T-72 lomps (descriptions or illustrations
_ _ "I:'rq, g!! !!Igr!:!!!9q9ry 9!9!!the !!s!1!!9 iqqlltvg i!t9rnq!!y9!-
To motivate contractors and accelerate customer action, Avista may consider creating a lighting
contractor partnership program, with incentives paid to contractors (or rebates paid directly to
t_ .plE{gllgtg!9o-yl9E'!ffy*gmggto qq!at{lq[!g!$E'_yMS ilgll_{v-et r9T1!tlE!1!h:
Avista should consider a new program, targeting replacements of T-12s in inventory to help
customers upgrade to more efficient new fixtures and lamps, and to move toward realization of
To ensure the recognition and longevity offocused outreach efforts, Cadmus recommends Avista
continue expanded annual market campaigns to enable more focused targeted marketing for the
nonresidential programs. ln addition, nonresidential programs may benefit from these additional
suSgestions:
o Develop o detoiled morketing plon enobling onnuol trocking ond ossessment of octivities,
The marketing plon would identify torget oudiences, clarify morketing objectives, ond
id e ntify evo I uoti o n m etrics.. Continue efforts to enhance the business website through promotions ond feotured business
Comolete
Complete
ln Progress
88
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 104 oI 127
information tools (such os Efficiency Avenue), testimoniols, general progrom brochures; ond
encouroge easier occess for trade allies through feotured guidelines ond tips.
c Track missing dota fields in Soles Logix, ond include these in extroct dotobases.. Document gA procedures or checklists to reduce missing or inconsistent dota entry.c ln oddition to checking for missing doto, Avisto stolf may benefit from developing o checklist
for staff entering porticipant data into dotoboses, ensuring all dato are collected
consistently.
Work toward integrating customer information tracking databases, thus enhancing efficiency and
lgqegl'qery9l._,
Consider incorporating changes to forms to account for new data collected through calculators.
QA ana Verification
ln Progress
ln Progress
! Prosrgl
Cadmus recommends Avista continue strengthening feedback loops for performance review of large
projects. To achieve greater consistency, Avista should consider documenting pre- and post-
' inspection protocols, which could include the following, recommended, industry best practices for
C&l plqgralns:
. . Estoblish inspection frequency, bosed on a progrom's relotionship with vendors, number of
vendors, types of measures, proiect volume, variability, and size of projects.
c Obtain a rondom somple ofvendorond measure types.
I . Clearly define pre- ond post-inspection policies and procedures.
. Require rondom, on-site inspections of 10% to 20% ol projects in lower-incentive prescriptive
progroms.
. t9q!!!s pLe-p!9i99! !!rp?4!9!s Io! C!! !9lC9p!9k9t; ,vvitlt hlgltlv un99ryoin bosetine conditions.
ln Progress
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
Khawaja, The Cadmus Group, lnc
Schedule 3, Page 105 ot 127
Appendix C: 2OL2 Non residential Process Eva luation Memora nd u m
This section provides the text from the nonresidential process evaluation memo drafted by Cadmus and
sent to Avista on August 2,2OL3.
To:
From:
Subject:
Date:
MEMORANDU M
Lori Hermanson, Avista
Danielle Kolp and Hope Lobkowicz, Cadmus
2012 Process Evaluation Memorandum
August 2,2OL3
Cadmus' 2012 process evaluation activities for the Avista nonresidential portfolio included the following:
c A Best Practice Comporotive Review (memo delivered in February 2013);
. ln-person interviews with program stakeholders; and
o Database and realization rate review.
Because Cadmus is not developing a formal process evaluation report for Avista until 2014, this memo
presents the findings of the staff interviews and database and realization rate review conducted for the
2Ot2 program year. Our objective is to provide key personnel at Avista with findings now to assist them
in improving program processes in real-time.
Key Findings
lnterview Findings: Large Project Review Challenges and Changes
ln August 2011, Avista instated a new internal system to independently review site-specific projects with
incentives greater than 550,000. This review stemmed from a recommendation in the 2010 Moss Adams
process report, pursuant to the 2010 Washington Utilities and Transportation Commission (UTC) rate
case settlement terms. The objective of the independent review was to examine project evaluation
reports prior to entering into contract with the customer, to ensure that:
o All supporting documentation was in place,
o Savings calculations were reasonable and well supported, and
o The project complied with tariff rules.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, Inc
Schedule 3, Page 106of 127
90
Avista staff who participated in the review process experienced multiple challenges, which are discussed
in more detail below. By the end of 2OL2, staff concluded that the review process was not functioning
efficiently, nor did it align with the intention of the Moss Adams report recommendation. Avista
suspended the review process on January t,2Ot3. ln 2013, Avista intends to implement a new approach
for reviewing site-specific projects, with the goal of balancing customer service and expediency with a
sound review. ln June 2013, Avista demand-side management (DSM) staff were finalizing this new
approach.
Review Process Challenges ldentified by Avista
Cadmus interviewed five Avista DSM staff who were involved in the review process. During the
interviews, we discussed several core areas of concern with the process and determined that the
intended protocol was not being followed. The process dictated that the Planning, Policy, and Analysis
(PPA) team independently review the energy savings and proposed incentive levels of all site-specific
projects with incentives greater than 550,000, to ensure these impacts were calculated reasonably. ln
2012, only one-third of projects that met the criterion were sent to PPA for review.
When Cadmus asked staff about the challenges with this review process, the following four main issues
surfaced:
3. Different focused ottention ocross teams. One staff person reported that the key personnel
within the DSM department involved in the review had different focused attention, which in
some cases translated to varying objectives for reviewing and approving projects. This is a
problem across many organizations and is, by no means, limited to Avista. While
implementation teams are most concerned with customer satisfaction and speedy and efficient
delivery, planning and evaluation teams are most concerned with compliance. At Avista, the
lmplementation team was focused heavily on the customer relationship, while PPA was focused
on ensuring compliance with the tariff, minimizing the risk of uncertainty associated with
claimed savings, and navigating relationships with regulatory bodies and stakeholders. This is
not to say that neither team was unconcerned with the othe/s objectives. While staff agreed
that their roles support the comprehensive functions and o// overarching goals of Avista's DSM
programs, specific daily priorities added to misunderstandings about the value of the review
and, in some cases, differing opinions on how and when to resolve issues.
Tronsporency. Some staff who were heavily involved in Avista's site-specific projects reported
not understanding the purpose, actions, or outcomes of the review. Without program-
stakeholder buy-in at all levels of the process, successful implementation was challenging. One
particular concern was a lack of information regarding how long the review would take to
complete for each project; this made it difficult to communicate accurate information to
customers on the status of their projects and the expected timeline.
Time log and time commitment. A common obstacle cited by all staff interviewed by Cadmus
was that the review process took too long to complete for each project. Often, the issues
identified during the review required further discussion to understand the assumptions behind
4.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 107 ot 127
the savings estimation, new data or information requests from the customer, or new analysis,
which caused delays. Another challenge was the volume of the projects and limited staff
resources. Having only one engineer dedicated to reviewing the large projects was problematic
and often caused bottlenecks.
6. Linking review with concrete octions. The review process lacked a formal follow through
procedure for problems uncovered during the review. This caused frustration as, at times,
findings and recommendations were not implemented. lnterviews and documentation of the
review process indicated that the extent to which the issues were resolved varied. For enhanced
delivery of DSM services, there needs to be an agreement regarding the best path forward for
calculating savings.
lssues Identified Through the Large Project Review
One of the major findings of the review was the overall reliance on customer-supplied data and the
need for a reliable and replicable approach to source that data. Avista staff were in agreement that
increasing the clarity and transparency about where engineering assumptions and inputs were coming
from was a needed improvement and a successful outcome of the review process.
Cadmus reviewed the communication logs for 22 projects that underurent the internal review. ln
addition to the above issue of reliance on customer-supplied data or assumptions (which was inaccurate
in some cases), the following issues were documented for these projects:
o lnteractive effects were accounted for incorrectly;
. Projects had missing documentation, such as invoices; and
. Engineering errors resulted in incorrect claimed savings and incentive amounts (the significance
of these errors varied in size).
Planned Process lmprovements
ln 2013, Avista staff worked together to design a new system to address the challenges cited and issues
discovered with the 2012 review process. The staff is currently implementing a two-step review process
for all site-specific projects that entails a technical review by the engineering team and an administrative
review by program staff.
o Technicol Review: Ensures that savings and incentive calculations in a project's Evoluation
Report arc well-supported, and calculated according to tariff terms and Dual Fuel lncentive
Calculator policy. The new system includes a checklist with questions that guide the review,
along with instructions and policy guidelines. The Technical Review will be completed before the
evaluation report is sent to the customer, which contains estimated energy savings and the
corresponding incentive level.
o Administrative Review: Ensures that minimum requirements are met before a contract is issued
with a customer and before an incentive is paid.
92
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 108 ot 127
ln the new process, PPA conducts random spot-checks to QA/QC projects, and ensures that the review
process is smooth and effective. A main distinction between the2OL2 and 2013 process is that this
random spot-check is intended to happen afterthe project has entered contract, or, in some cases, after
the incentive has been paid. According to implementation staff, this will help overcome bottleneck
challenges.
Both checklists (the Technical Review and Administrative Review) will be formalized documents known
as Top Sheets, which will be attached to project documentation through the life of the project. Avista
intends to synchronize the Top Sheet information with Tracker, the engineering database, and with
SalesLogix, the customer information system that houses nonresidential rebate and incentive
information. ln June 2013, the lmplementation team began using Top Sheets for all projects.
2077-2072 Datobose and Reolization Rote Review
As part of the 2012 process evaluation, Cadmus reviewed Avista's 2012 nonresidential project database
and the 2011 and 2OL2 realization rates for the nonresidential portfolio. The documents that were part
of each effort and our associated research questions are listed in Table 39.
Are data being tracked accurately and consistently?
Database Review 2012 SalesLogix
Database Extract Are contracts issued in accordance with Avista policy?
oo inEntives compty witfr tiriff ruteiloiwishinston and ldaho?
2OLl and 201^2 Why do some projects have a very low or very high realization rate?
lmpact Evaluation eie tnere opportunities for lvista to improve the process of
Sample calculating reported savings to improve the realization rates?
Realization Rate
Review
Table 39. Database and Realization Rate Review Activities
9i
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 109 ol 127
Database Review
Tariff Schedules 90 and 190 govern how Avista can spend funds from the Energy Efficiency Rider
Adjustment paid by Washington and ldaho ratepayers." To assess compliance with these Tariff
Schedules, we examined two main indicators:
1. Project incentive amount: electric and natural gas project incentives should not exceed SWo ol
the incremental cost of the project (p. 3 of Schedule 90; p. 2 of Schedule 190).
2. Project simple payback.
a. For lighting measures, the simple payback period must be a minimum of one year and
should not exceed eight years. (p. 2 of Schedule 90).
b. For non{ighting electric and natural gas measures, the simple payback period must be a
minimum of one year and should not exceed 13 years. (p. 2 of Schedule 90; p. 2 of Schedule
190).
The tariff rules make exceptions for the following programs or projects (p. 3 of Schedule 90; p. 2 of
Schedule 190):
o DSM programs delivered by community action agencies contracted by Avista to serve limited
income or vulnerable customer segments, including agency administrative fees and health and
human safety measures;
o Low-cost electric/natural gas efficiency measures with demonstrable energy savings (e.g.,
compact fluorescent lamps); and
o Programs or services supporting or enhancing local, regional, or national electric/natural gas
efficiency market transformation efforts. (ln 2012, Avista considered new construction fuel
conversions in multifamily building projects and T12 to T8 commercial lighting conversion
projects as market transformation efforts.)
Applicobility ol Tarilf to Presuiptive Projects
At the time of this memo, Avista's tariff was undergoing revisions and a new tariff was filed on June 26,
2013.
Avista uses the tariff provisions to: 1) design prescriptive measure offerings and incentive amounts and
2) evaluate the eligibility of site-specific projects on a project-by-project basis to ensure compliance
before approving them. Cadmus does not believe the tariff language was clear enough on the topic of
22 Schedule 90: Electric Energy Efficiency Programs, Washington, Available at:
htto://www.avistautilities.com/services/enerevpricine/wa/elect/Documents/WA 090.odf; Schedule 190:
Natural Gas Energy Efficiency Programs, Washington. Available at:
http://www.avistautilities.com/services/enersvoricins/waleaslDocuments/WA 190.pdf; and Schedule 90:
Electric Energy Efficiency Programs, ldaho. Available at:
http://www.avistautilities.com/services/enerevpricins/idlelect/Documents/lD 090.pdf
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 110 ot 127
94
compliance to conclude whether individual prescriptive projects should be subject to the simple payback
period and incentive cap restrictions at the time of rebate application approval. lnternally, Avista staff
also expressed disagreement on this matter.
For purposes of this review, Cadmus evaluated both prescriptive and site-specific projects against the
provisions of the tariff described above, to allow Avista to review the findings and incorporate them into
their planning. lt should be clear that by presenting the prescriptive findings below, Cadmus is simply
suggesting that better clarity is needed and not necessarily that these projects were out of compliance.
Avista's proposed tariff clarifies that moving forward, site-specific projects are subject to the incentive
cap and simple payback periods at the time of project approval, while these parameters will be used in
the planning process for prescriptive measure offerings and incentive amounts.
Simple Payback Findings
The majority of projects were in compliance with simple payback rules. Cadmus found that all site-
specific projects met the 13-year and eight-year payback periods, with the exception of some legacy
projects that were initiated before the new tariff rules took effect on January t,ZOLL.
Less than tO% of prescriptive projects exceeded tariff simple payback periods. Table 40 summarizes our
findings.
_si!:_s! g.t ri! rygr " !!l
Prescriptive Lighting
(includes market
transformation and T12
projects)*
Prescriptive Non-Lighting
(excludes multifamily)
L3% s8ss,s3s4,438,942 kwh
iis,ees rwh' '
2%
572,7377,810 therms 7%
Upon reviewing a sample of 10 prescriptive lighting projects that exceeded the eight-year simple
payback period, Avista found that five projects involved a T12 to T8 conversion and three projects
contained database errors that inflated the simple payback period. ln these cases, what should have
been entered as months were assumed to be years, and multiplied by 12.
The sample size for this manual review was not large enough to extrapolate findings to the full
population. However, based on the review findings, it is probable that a large proportion of the projects
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 111 of 127
Table 40. 2012 Projects Exceeding Simple Payback Periods
included in Table 40 involved T12 to T8 conversions and/or experienced database errors, thus
significantly lowering the impact on energy savings and incentive costs.
Project lncentive Findings
Site Specific
The vast majority of site-specific projects had incentive costs that were compliant with the tariff rule not
to exceed 50% of the incremental project cost. lnitially, Cadmus found 74 site-specific projects (19%)
that exceeded this cap. Upon reviewing these projects, however, we found that nearly half experienced
a rounding error from Avista's Dual Fuel lncentive Calculator that put them over the 50% limit by just
SO.ZS lsee Figure 43). Avista staff reviewed the remaining projects to understand why they exceeded
the incentive cap, and found that the majority were incorrectly entered in SalesLogix. Avista reported
that these projects had been calculated and processed as prescriptive projects, but incorrectly entered
into the database as site-specific.
Figure 43. Range of lncentive Amounts Exceeding 5oo/o of lncremental Costs, 2012 Site-Specific
Prescriptive
Significantly more prescriptive projects (74%ol exceeded the 50% cap. As noted above, this finding was
expected because Avista's program design and delivery strategy did not consider prescriptive payments
as being subject to the tariff rules, and the lighting market transformation effort exceeded 50% by
design. Table 41 outlines the incentive payment and energy savings impacts from projects that exceeded
the 50% incentive cap.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 1'12 ol 127
Projects
S10-S100 S100-Ss00 Ss00-S1,000 S1,000-Ss,000 overSs,000
50
50
-4oE:oL't30oot20
10
0
Under S10
96
Table 41. 2012 Prescriptive Proiects Exceeding 50% lncentive Cap
Prescriptive Lighting
(includes market
transformation and T12 2,574 80o/o 26,747,965 kwh 8Lo/o 52,29O,O3L
Prescriptive Non-Lighting
(excludes multifamily)
3,220,70E.kwh 58%
16,684therms | 14%547s,437 45%
I Total Prescriptive
29,968,569 kwh 77%
74%15,684therms L4%2,923 52,76s,468
* Cost impact represents the aggregate amount exceeding 50% ofthe incremental cost.
** Avista's database extract does not denote which projects involved T12-T8 lighting conversions.
Again, Avista manually reviewed 10 lighting projects that were over the 509/o cap, and found that eight
were T12 to T8 conversion projects, considered market transformation. Based on these findings, it is
probable that a large proportion of the lighting projects listed in Table 3 involved T12 to T8 conversions,
which would greatly reduce the cost impacts and energy saving impacts of from lighting projects over
the 50% cap.
Datd Entry ond Dota Tracking
ln addition to assessing policy conformance, Cadmus reviewed the 2012 database for data accuracy and
completeness. We found that:
o 8 projects were recorded as paid before construction was completed (most of these were entry
errors)
L2% of all projects were missing Construction Complete dates
44 projects (l% of all projects) were missing incremental cost data
o LSYo of site-specific projects were missing contract date fields in SalesLogix
o 44Yo of site-specific projects were missing post-verification dates (and it is Avista's policy to
conduct post-installation inspections of all site-specific projects)
Avista reviewed 20 prescriptive lighting projects to determine whether they were market-
transformation projects (as noted above). They also uncovered several data errors with these specific
projects. ln all 20 projects, at least one of the following issues was found:
o Simple payback periods were entered in the database in years instead of months,
o Simple payback periods were entered incorrectly (Saleslogix data fields were not consistent
with calculations),
o Prescriptive projects were entered as site-specific projects,
a
a
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 113ot 127
lnformation from invoices regarding quantity and type of light fixtures was not transferred to
prescriptive incentive forms and SalesLogix correctly,
lneligible measures were rebated, and
lncentives were calculated incorrectly.
Realization Rate Review
Cadmus' impact evaluation methodology consisted of validating the reported savings for a sample of
projects by conducting independent metering, simulation, or regression analysis and by visiting the
project sites to verify that equipment was installed and operating as intended. The result of our project-
level measurement and verification tasks is a verified, ot ex post, savings value for each project in the
sample. The ratio of verified savings to reported savings is the project's realizotion rote. A realization
rate of 100% indicates that no adjustments were made to the reported savings value.
ln 2011, Cadmus' nonresidential impact evaluation sample consisted of 179 electric and gas projects.23
Of those , the majority (n=112) required a saving adjustment by more than 10%. That is, 63Yo ol projects
had realization rates of either 110% or greater, or 90Yo or lower. Specifically, just 35% of electric projects
and 42% of gas project realization rates ranged between 90% and 110%. This changed in 2OL2, when the
majority of projects (54 of 101)24 experienced realization rates between 9Oo/o and 110% (see Figures 4
and 5 below).
Cadmus analyzed how frequently the evaluation resulted in an upward or downward adjustment of
reported savings, by how much, and the reasons behind the discrepancy between reported and
evaluated savings. The purpose of this review is to provide Avista with information to assist in improving
the reliability of the reported savings in the future, thereby improving realization rates for the
nonresidentia I portfolio.
Direction, Frequency, and Magnitude of Verified Sovings Adjustments
Cadmus determined that when savings needed to be adjusted by more than 10%, they were more likely
to decrease than increase. ln other words, most reported savings for projects in this group were being
overestimated, and the verification process resulted in a downward adjustment. This was true for all
2011 projects, and for all 2012 electric projects. ln 2012, gas projects required more upward
adjustments.
This number reflects projects with gas savings and electric savings. We actually evaluated 157 unique projects,
some of which achieved dual-fuel savings. For the purpose of the realization rate review, we treated gas
savings separately from electric savings.
The full 2012 impact evaluation sample contained 109 projects. We excluded eight projects from our analysis
that still had measurement and verification activities occurring at the time of writing this report.
a
a
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 114 ot 127
98
2011 Projects
Figure 44 illustrates the distribution of realization rates in increments for 2011 projects. ln 2011, 51
electric projects had a realization rate below 90% (42%l,while 27 electric projects had a realization rate
above 110% (23%). Gas projects exhibited a similar pattern, with 25 projects having a realization rate
below 90% (44%) and eight having a realization rate above LtO% (L4%).
Figure 44. Distribution of 2011 Realization Rates by lncrements for Electric and Gas Projects*
*Note: Percentages may not match above text exactly due to rounding
For electric projects, the relative proportion of reported kWh savings in each increment was relatively
consistent with the number of projects in that increment. However, for gas projects, the relative
proportion of reported therm savings in each increment did not accurately represent the corresponding
number of projects. For example, while just t9% of gas projects experienced a realization rate of below
50% (but more than 0%), these projects represented 47% of reported savings.
Dividing the projects by increments revealed that a large portion of the projects with realization rates
below 90% were in fact below 5Oo/o, and most of the projects with realization rates over 110% were
actually over 150%. This indicates that not only was the range of realization rates large, but a significant
portion of reported savings values were substontiolly different from verified savings, requiring an
adjustment of 50% or greater.
2012 Proiects
ln 2OL2, realization rates improved. Rates were less variable, and projects required smaller reported
savings adjustments than those in 2011. For example, 51% of electric projects and 57% of gas projects
had a realization rate between 90% and 110%, leaving only approximately one-third of projects that
required an adjustment over 10% (see Figure 45).
Exhibit 3
Case Nos. AVU-E-l4 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 115 ot 127
Electric Projects (n=120)
Proportion of Reported kWh
Gas Projects (n=59)
Proportion of Reported Therms
OYo LO% 20% 3OYo 40% SOYo 60% 70% 80%
r0 rBelowso% 50to75% I75to90%
r 90 to U0% r 110% to 725yo r125%to LsO% t Over 150%
99
Of the 2012 electric projects that required an adjustment over tOTo, most required a downward
adjustment (18 projects; 31%). This is consistent with 2011 results. Of those 20L2gas projects that
required an adjustment over 10%, the direction was upward (eight projects; L9%1.
Figure 45. Distribution of 2OL2 Realization Rates by lncrements for Electric and Gas Projects
*Note: Percentages may not match above text exactly due to rounding
Cotaloging Projects with High and Low Reolizotion Rates
To understand more aboutthe projectsthat had severe adjustmentfactors (very high orvery low
realization rates), we conducted a desk review of the project files and engineering analyses for a sample
of projects from 2011 and2072. Specifically, this sample entailed projects with electric savings that had
been adjusted by over 25% in either direction (a realization rate below 75Yo ot aboue L2SYol.
The original sample size was 75 projects; 57 from 2011 and just 18 from 2Ot2. Upon reviewing the 2011
project files, we found that seven projects did not have sufficient reported savings documentation to
accurately conclude the reason for the savings adjustment. Therefore, the final 2011 sample size was 50,
leading to an overall sample size of 68.
Based on our review, Cadmus concluded that there were nine main reasons for the savings adjustments;
these are outlined in Table 42.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 116of 127
Electric Projects (n=59)
Proportion of Reported kWh
Gas Projects (n=42)
Proportion of Reported Therms
o% Lo% 20% 30% 40% so% 60% 70% 80%
r0 r Below 50% 50to 75% t75to90%
r 90 to 110% I 110% to 125%.L25%to 150% I Over 150%
1@
Table 42. Reason Categories for Variable Realization Rates
1. Participant Operator Error
2. Calculation Error in Reported
Savings
Savings required adjustment due to customer actions, such as installing or
operating equipment incorrectly
Reported savings calculations or assumptions were incorrect
3. ENERGY STARo Appliances
Deemed Savings Update
4. Cadmus Metering Results vs.
Avista Simulation or Analysis
5. Cadmus Metering Results vi.
Avista Metering Results
6. Database Error
7. Cadmus Calculation
Methodology vs. Avista
Calculation Methodology
8. lnaccurate Lighting Hours-of-Use
(HOU) Estimates
9. Equipment Verification
Cadmus used updated deemed savings values for ENERGY STAR clothes
washers, dishwashers, freezers, and refrigerators to verify savings,
requiring an adjustment from the reported values, which relied on older
deemed savings estimates
Cadmus used metering results to inform verified savings, while Avista used
,other to_ols to generate !:ported TyilCs estimates
Both Cadmus and Avista used metering results to inform savings values;
however, the companies' parameters or timing differed
Some values in the database extract were erroneous due to a database
error, not a human error, and savings needed adjustment to reflect the
accurate value
Cadmus and Avista used different methodologies to calculate savings (i.e.,
regression analysis versus simulation), creating different results
The reported savings for some lighting projects were based on incorrect
. HOU assumptions
The on-site equipment parameters (size and efficiency) differed from the
assumptions used in the original savings estimate
f n 2011, the most frequent reasons for savings adjustments of 25% or greater were due to metering
results being over the original estimates formed using simulation or analysis (n=10) and calculation or
assumption errors in the reported savings values (n=10). Other top reasons included ENERGY STAR
deemed savings updates (n=9) and differences in Cadmus' and Avista's calculation methodology (n=8).
ln 2OL2, there were far fewer projects with adjustment factors of 25% or greater. The top reason
categories in 2OL2 stayed relatively consistent with those in 2011, excluding the ENERGY STAR deemed
savings updates.
Figure 46 illustrates the number of projects in each of the reason categories outlined in Table 42, across
both years. Table 45 at the end of the memo, lists the specific projects included in the review and a
description of each project's specific savings adjustment.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 117 ot 127
Figure 46. Number of Projects with Savings Adjustments of 25% or Greater by Category, 2OLl-2Ot2
lmpact on Gross Savings
While the majority of savings adjustments in 2011 resulted in decreased savings, certain reason
categories experienced realizatlon rates higher than 100%, on average. For example, three reason
categories (Cadmus Metering Results vs. Avista Simulation or Analysis, ENERGY STAR Appliances
Deemed Savings Update, and Equipment Verification) resulted in increased savings. ln other words, the
projects in these groups experienced realization rates higher than 100%, on average.
ln20L2,just one reason category (Cadmus Metering Results vs. Avista Simulation or Analysis) resulted in
increased savings. Projects in the other 2012 reason categories (Calculation Error in Reported Savings,
Cadmus Calculation Methodology vs. Avista Calculation Methodology, and Participant Operator Error)
resulted in decreased savings.
The aggregate kWh impact for each 2011 reason category is listed in Table 43. The aggregate kWh
impact for each 20L2 reason category is listed in Table 44.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 118 ot 127
L2 10
:z
..""-" "'d-rf c"'1.*'"]-""-*$$"d
$*uo'
.2072 12077
102
:s eNE,ir ---'= Y o-hr=sg
t9o-?o)+E838-{ o-9,;5 to'-6a.6E0, 6.x
EE-Y
a
X :R>R S X:RNFdHOsi
a xx :R s;RoooooFd
xxs;Rma\cro
-,-,:..-.
H6d6l@6HttNmNAilnNNd(ontro
^iodlal,ll\NmN@60HN(oO600N6'ro@t\lDOui€Nt\rHl,ld
o,
l!
@
(9
o
r}N
o
c
OJ
E
:,
E'
0!c'tl!ta
.:3
Io'6.
o-
o
0,
oE!q,
r!(J
tro
(!od
.E!Eo
IE
Io
00c.;
(!th
o
.E
o
!c(!
1'(u
o
CLod
r{o
di<foEl!F
sao
;3 :RNOc;
o;
CL
E@
C
.9
G
=G
o
IoTL
.Eo.c
.;
@'to
Eo'-o
Eo
o
o
coooCLo
ooaEoo
E
ocos
.9
.gsF*
<trN6at<fmoN6l tf6o6l
x sa ;R s>RNFddodN
N 'Cn e si I 6N ,d r.o lJ.) (rr+rNmoN@60r,r(D€f\tOONNIHrr1 N
oodod6<tNod6
di .i Fa jjNdN6dHrt(o +
xo
6d(oo@HOsfOdNF
^ioiioi6N<lOr
60rrNrodr)@NNINdoiFiN6hmo
:RSr\O
N<toromm(orcon@odF:.iomod@H
6
'
Nddo ut
q
EEc8=.9Eoo!0d.9.= @=9q'=e,igS;
",>qJc; -5I$Ec i=I I i,9r€
F.mr\dorr+ oo'oom rN
(o(n
\o6of l-oO'no-Eoo I
=CG6= EE=- o*C.,6-g6cEggE
=dE.eU9.=PE9;E(J:UC
mor
OF.i risfd
^-odl@.1
ln
o00
i* s,Y_>i=* b <iaEs:.Et>'>-!2bo:o60ro(J=iUSE
rruE;i:oc;io(Jz
l
mdr\<fln
ooN
FFm
o
CN6(Oloir@dldi
6(rorNIdirroltt (
o(
3HE!aE
=c!+E<&EEcm-'-.=ooo=sPIU
=Eb3ur€-G:
=6o6>6u<(J
OJac€
oo
G,
Fq
soCG,'=u
G16U
EiEF
n=EE
4EF38-{ o-9u;E*o'-irz,6E
H93o-EY
a
voFl
6Hm|,i6
:Rst
o
r!o
(9
osln
o
Eo
E,E
b0c'=
(E
s.=3
qoo
A
o
(IJ
otoo
IE(,co
l!oG
.=31'o
.gg0
bo'tl!
!,o
.E
o
EcG!,o
oCLoG,Nr{o
{<lo
=tE
' loilhllo
dro
'---X:Rol6
.lit
ol-o idl lo Hi
Nlh,'lq
.l, tnlIro ,Ldt;ir.a I
o,
Li.o oLcr hl6ts
d6NF tt
rYi
r f-lelold
l
li1
lil-iol l
L!t, joisi Iolo-1 Ioi
cr6Llo Ii'E - Ll'E G I
GOo-F
g
CL
EG
co
of
E
q,
aGo
.EoE
.s
EOc
G
t,o
'tro
Eo
0,
o
coIoCL
G
oooCoo
E
oco
.3
.2
F*
NNN6N(o*i diNOr
slR<tH
O6lNO)do+diMN6d
st(!N(ort I\oo' d(o<fom
- -'1;-
rnOONjd
ON+- si
d
ro -
uoE!t!2(lja9iooa=ozcoccoo'E .=E6i!= oE* aEo(Joc(JoE.93ETEcrl4==c
=qeO.=H;9GdU'ZU6
HIsfNoiorsfr
I
-i
:RN
L
mNoflm lsff
mlr\idl
6l@N-ii
-J
H:d :N++Lrnlj
i
-i
@
I
6-!4> oloil:c
=c8SE 6iu -l.= olg6
3 ErI6lI oiE9E.9 IG>L)<
Figure 47 illustrates 2011 projects in each reason category as a percentage ofthe total sample compared
to the percentage of each categories' net kwh impact. While the ENERGY STAR Appliances Deemed
Savings Update category contained nine projects (representing about 8% ofthe total sample), the net
difference in ex onte and ex post savings was actually minimal: a gain of 1,151 kwh (see Table 43), less
than 0.07% of savings in the impact evaluation sample. The Cadmus Calculation Methodology vs. Avista
Calculation Methodology category had similarly minimal savings despite containing a relatively large
number of projects (eight). On the other hand, the Cadmus Metering Results vs. Avista Simulation or
Analysis and Participant Operator Error categories representedS%and 3% of projects, respectively, but
the net differences in ex onte and ex post savings represented 13% andT% of the total verified savings in
the impact sample, respectively.
ln 2012, the percentage of projects in each category was higher than the respective percentage of kWh
savings in each category (see Figure 48). For example, the Cadmus Metering Results vs, Avista
Simulation or Analysis and the Cadmus Calculation Methodology vs. Avista Calculation Methodology
categories both represented LO% of all projects in the evaluation sample, but their net differences in ex
onte and ex post savings were relatively small, representing only 2% and 4% of the total verified savings
in the sample, respectively.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 121 ol'127
Figure 47. Relative Proportions of Projects and Savings lmpacts by Reason Category, 2011
Net Difference as % of Verified Savings in Sample
% olTotal Projects in Sample
I Metering vs. Simulation
ES Appliances Update
I lnaccurate HOU
, Database Error
I Equip. Verification
5% L0% 75% 20% 25o/o 30%
r Calculation Error, Rprt'd Savings
I Diff. Methodology
r Participant Error
r Diff. Metering Results
105
Figure 48. Relative Proportions of Projects and Savings lmpacts by Reason Category, 2012
Net Difference as % of Verified Savings in Sample
% of Total Projects in Sample
O% 5% 70% 75%o 20% 25Yo 30% 35/o
r Metering vs. Simulation r Calculation Error, Rprt'd Savings Diff. Methodology r Participant Error
Conclusions dnd Recommendotions
Based on the above findings, we offer the following conclusions and encourage Avista consider the
recommendations listed below to improve their internal processes.
large Project Review Process
Conclusion: Avista's 2011 Large Project Review process was not implemented successfully due to a
series of communication issues and the absence of a mechanism to address concerns about project
parameters and correct mistakes. ln the first half of 2013, Avista has been designing a new process for
all site-specific projects. While this process is underway, we have several recommendations may assist
Avista with successful implementation and an effective process.
Recommendations:
o Eflectively communicote the new projed review process to oll key team members. Many of the
issues identified through Avista staff interviews regarding the prior review process centered on
communication challenges. When implementing the new process, ensure that all stakeholders
have a clear understanding ofthe review goals and correct protocol.
o Ensure there ore cleor protocols in ploce for oddressing issues identified during the review ond
the spot-check. To ensure that Avista and its customers are benefiting from the time and
resources dedicated to this process, consider implementing some check-points and policies to
clarify how and when to alter project savings and incentive levels if issues arise during the
review. This may include designating a senior-level point person to serye as the decision-maker
for questions or disagreements regarding a project or its calculation methodology. Consider
identifying methods to ensure that all issues are discussed and resolved before incentive
amounts are communicated to the customer.
706
Exhibit 3
Case Nos. AVU-E-'|4 AVU-G-l4
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 122 ot 127
Estoblish o goal lor the number or percentage of projects that should undergo o rondom spot-
check. Avista's new process dictates that the PPA team will independently review a sample of
projects, in addition to the peer review process. We suggest establishing a clear metric for the
number or percentage of projects this sample will include, such as five projects or 10% of all
projects.
Estsblish o reasonable gool for how long the review process should take. A core challenge with
the prior review process was the time lag. Keeping in mind that any process aimed at improving
the quality and accuracy of incentive payments and claimed savings will add time to existing
procedures, Avista should internally discuss the amount of delay that is reasonable. lt may be
beneficial to create objectives for how long various steps ofthe process should reasonably take.
For example, Avista could establish one goal to complete the first Top Sheet review within a
certain timeframe, then establish another goal to guide how long it should take to resolve any
issues, if identified.
Consider odopting o tiered opprooch to the review so thot lorger, high-risk projects receive
more scrutiny before contracts qre issued ond incentives ore paid. Under the planned
approach, all site-specific projects will undergo peer review. Often, utilities employ a risk-
mitigation approach to ensure that the largest and most expensive projects receive the most
rigorous review before they are approved. Avista might explore adjusting their review process to
focus the most time and resources on larger projects. An example of this type of approach is
provided in Table 45.
Second Engineering Review : Projects above 550,000
Third Engineering Review Projects above S75,O0O
PPA Review Projects above 5100,000
Pre-lnstallation Visits Projects above 5100,000, plus others as needed
na1!91 auait (spot-checkl s oroi991 or t 0! ofall projects
o Consider structuring rondom spot-checks, or "oltdits," to occur ot vorious times of the process.
The current review structure plans to have some projects receive independent review after the
project evaluation report is complete or after the project is paid, so that any mistakes can be
corrected for future projects. However, it may be beneficial to stagger projects so that a
random portion also receives independent audits before incentive information is communicated
to the customer.
Database and Realization Rate Review
Conclusion: The accuracy of Avista's claimed savings, measured by realization rates, improved
significantly from 2011 lo 2Ot2. Three of the four main reasons for large savings adjustments in 20L2
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 123 ot 127
Table 45. Example of Tiered Approach to Large Project Review
are largely outside Avista's control. However, Avista can still improve the reliability of claimed savings
estimates falling into the reason category of Calculation Error in Reported Savings.
o Recommendqtion: Continue to move forward implementing the new review process to identifo
and resolve savings calculation errors.
Conclusion: Most of the nonresidential projects were compliant with the 2012 tariff rules, but
disagreement among DSM staff on tariff interpretation makes it difficult to draw conclusions about
prescriptive projects. Avista has already begun updating the tariffto address this concern and create a
more coherent policy. There are several improvements Avista can make to data tracking activities to
clarify policy compliance on a project-by-project basis and improve data collection overall.
Recommendotions:
Cleorly document legocy projects or mqrket tronsformation projects in Saleslogix. Avista's
tracking system specifies measure type, but lacks detailed information such as whether the
project involved a T12 to T8 lighting conversion. This makes it challenging to understand which
projects are considered market transformation. Further, legacy projects are not specified. To
streamline internal tracking, auditing, and evaluation, consider adding a field to denote which
projects are eligible for transition policy (legary projects) and which projects are considered
market transformation, as well as any other project characteristics that warrant exception to
tariff rules under Avista's new policy.
Continue to improve doto entry in SalesLogixto reduce missing or incorrect fields ond enhonce
the comprehensive ddtoset.
Exhibit 3
Case Nos. AVU-E-14 AVU-G-14
S. Khawaja, The Cadmus Group, lnc
Schedule 3, Page 124o1 127
708
c/rt oF-
-F C:N
nrir -F'= Y cL!t+ J e
'=EgtqoE?CDq€[38-{oo.E f<2FE_!l -oz,6Eo '6',9
EE-Y
@
o\oFl
oooPFPoooEEE'CL CL CLgglffOOq66>FOOOfi'nEPE
OOOGG66:=:66
tEoEodoE88nEr
qq6qq66quumuummMccccEccc
oGGEOOOOrhthqoqqqqppppppppttrtttttsCLo.OCLCLCLCLCLG,d,d,d,EG,EE
oooooooo
uuuuuru@c c E E E' C C C.9 .9 .9 .9 .9 .9 .9 .9
GGOOOOGOfflr:rfJfeeolllleJlll
o6GGOOG6(J(J(JUU(J(J(J
E0 hoEC'5 'too
t,E
o"ad,G
oo
UUtrCoo
6GfJ99
GOU(J
uuh0@@00@h0oooooooo6E66EE6E!Tr!'o!,!rt,Eooooo,ooo
=EE-C3<<SPPPPPoocrooootu
i5Ei5i5i5i5i5i5
H E x N H E H E il $ H N $ s $, s H U H $ H il H
Is* fl * E r3.[E.$- $' x' $B*xs * +
xsx at +s[$$*u :-iu gsgE$= s :
'=oo
ZqqxO=G_u 7i7.=oYo >e<EELE<Elooruoo60J.=.:-LUOUq,€€9g>EE-666e.q.q=*666rI!r-EEHd--=uuq
oLqEE3.E 55o6Eo.g.g-E99@iEooECcc(s - 'Ft-aa ::8ag E EoH88 E.b€8o.=JJOOVVlO
-'3oo^L^Looo^LS P8E-E.EEE -e.-s.-sEErrE-€3-€EEiirEEts:c;fiaa=aii-Esaddic
6 d r\l N <t d o 6 o d or N d @ ln 60 <i n rl) t 6 o6 @ CO N f\ l^ N m f\ @ N lO sf N d mst rn O) O) N r rco (o ro m or (o @ @ d <i d N m (o (Y) (nur (o m (n o o o@ sf st h (o rO n Ur |o O O) t (o (l) (o oo N r\ r\ r\ F Nm m f) m m m m m m d N m m m m Nm m m ff) m (n (n
ddddHHHdddHHiidiiddddddddddHHHddddHdidHHdiHdddoooooooooooooooooooooooNNNNNNNNNNNNNNNNNNNNNNN
l
EO
'&
oEPl!(J
3.g
oc,oPft,.E9bEE&.8bEsG,tr)cNeLooPtlc=9EECPq6
.=9EO(o '6,
OLoo o-c.'; r!)G<t6QJ(o=eet.=F.=fuo
ao'6'
-x5EB\fgB
a.g
o(o
qr=EEH
o
(o{ oi\sF c(\,lE,lr---
fi*EE
*9s4E838";{ o-19
9:E
&E'Y
U)
oFlFl
trcccccccEEo o o o o r4.g.o.o.g.g.g.g.g.g.gEEE'E!PPiu iE i! iE iu = G o o @ 6 6 O 6 o oET'Et'T'c6==========cLocLcLELOru=7=-======Lr r r r r E -.E.E.E.E.E.E .E.E.E.E = r r r r r e e o;;;; ; n uia in 6 6 6 6 6 6o o O O O O O O t t t9i 9{ 9l 91 9{ E.S,;,; 6,;,; 6,i q,;,; - - r - r - u u u
EEEEEgt SePPBSPHPPt x i x x t AEEE -o. -o. -e cL -. < .= .= .= .= .= .= .= '= '= '= = = = = = :cL cL cL e cL cLa o o o o o o o o o o o o o u o J:s:s:s6 6 6 q o o E;:
=:5 5 5 5 5 5 ! ! E ! ! ! e S EU U U U U U " ---:
(,
oEo.f
o0c.gEo.
qU
$xsdHH$nn$$HHHHHHE$H*$F$EEE
N N il $ i$*st$n*[$g$3s$sn$ - --i
R s s H s R * r t a n s g g ar +r $ g t B * n a $$
6 @ H d o 6 + O dr O o <l <f N F d @ @ F O O) r\ (o r\ 6 aOOci H N c @ N 6 st 6 d o m N (n N N f\ <t rn f\ l,) @ d d f\ tsmm o 6 6 6 O 6 d oO !n rO m t <t <l <t <t rO i + ln d O O (n NsiN ('c, h h N N rO rO (n (n sl m m m I\ l\ N ('r (l) (.c) (o N (o (o d dram m m m (n i1 m o d) (n (n m m m m m m N m m m m.m m m Nm
iln
oooocc U
UOU'=!'-.-'-mqeieEd=.=L.=Oo=-o.;egEd
I E I I !- O,I r- E { E 6:&iii3>_,_l____
oo
oocc
n0 !0ccE.:
.!9 .90JJlu lu
CL CL
oo
oo
AC
obo 'ico
OC,YMC?E-=
(J
I6o
co(J
oo
()(J(J
-EEII!oo.l4l2EEEEEctrtrcc€ss€Po u (J oio
E f f sis
o'
6:ol
=o-c9(r
tr>
c6-!ox,5 t.E qcFct€E!,a-Eox
-E.'S I 8 ,j
bfic.rr.rbPESSEa::rIe,-:-l
-1
oo=.cGE3=
oo=Eoo(J (J
€o
bO EDooECUU
oooE.c=ooG
==3st<.9 .9 .9aoo
GOG
uEo@o0Ja)cccUUU
dHHHdddHdddHHiHHddddHdddddHil ; ; ; ;i;; J J Jd i H d H i e I H iHdd dd d doooooooooooooooooooooooooooNNNNNNNNNNNNNNNNNNNNNNNNNNN
os
o
=o
oo
G
gocq
=;€eSi5e55;;oooee= gggeg=ees
(frsl ANs- trNE,L---'= Y o.=fr=EN
t(,oYlo)+E838";{ o-9aEPo'-6z. 6.c.
893ti -EY
a
"l".iH
66666bo00hobo@ccccE'5 't 't 't '5
6600Gq0qq6-E!rE!PcEcccctttEt.9.9.9.9.9.92*'g-o-oEE!EEEahabil-.cc--f=fff=o o o o o o o o'b b E E E .E E E b6 6 6 -6 6 = i ! t = i; ti,n ti ti t3
=! EP!! r u r u u "i d "i ri d ri uOOOOQcccccrrrttt!E E E € 5 .9 .9 .9 .9 .9 oo u oo oo oo oo Ee e e e € E E ts t E 5 E.='=.=.= ii
ESSEE****-HEEEEEEEoooo6uuuuu2zzzzzL
$$5$Hs$EE$E$$$$5s
t$"*tE*"n*qa$sng"
RsxEREilsi{SeflErilE
S s s ;' $ n' d ;' E H 3. i E H 6 E f
N + 6 N r @ 6 Or O + N (n lO (l, O H (oo i m o 6r \o or d o ot r\ (o m F 6 H N@ O 6 o) m f. g) d O F (n Ol N N N <t m6 m o 6 6 o <f cn h ot lrt cD @ 6 ol ot Nm m m m (n sf dt m m m m d) m m m m m
NNNNdNNNNNNNNNNNNJd:i.idHddddddHdHddooooooooooooooooo-NNNNNNNNNNNNNNC!N
om6
.=d
G.9'=JPor3 I3=-',..--€-:rE t- Hg==;38. EsEE.= ;b _E_EEUP€i
EsgeEgEgE=EEEEEEg
= U > @ q > o o'! = : :.9P-g)J r e2 < - ct c: I < (J 3.^o = = J J (J (J (J
a)a6q666-y66q66666l; - l; a a ui ui 6 = 6 ui 6 6 o c a a
':l
====B==3====3ts=3=