Loading...
HomeMy WebLinkAbout20240801Exhibit No. 4 - NEEA Response to ADM Evaluation.pdf May 4, 2023 >> neea To: Theresa Drake, Customer Relations and Energy Efficiency Senior Manager, Idaho Power Nicole Hydzik, Director of Energy Efficiency,Avista Utilities From: Becca Yates, Executive Director and NEEA Executive Team CC: Executive Committee of the NEEA Board of Directors Thank you for the opportunity to respond to the data request dated April 7, 2023. We appreciate the importance of the Idaho Public Utility Commission's questions and ensuring the Northwest Energy Efficiency Alliance (NEEA) is delivering value back to Idaho ratepayers, commensurate with their investment in NEEA and in alignment with NEEA's purpose and regional role. NEEA's executive team carefully reviewed the evaluation findings and recommendations and included a detailed response below. We shared the evaluation report and data request from Avista and Idaho Power, along with our response, with the Executive Committee of the NEEA Board of Directors. We respectfully request this response letter be submitted in full to the Idaho Public Utility Commission. This memo is structured in two parts.The first, addresses the recommendations put forth in the letter from Avista Utilities and Idaho Power dated April 7, 2023, and the second, is NEEA's detailed response to the Evaluation of NEEA Impacts Allocated to Idaho Power Company and Avista Utilities Within the State of Idaho, dated April 6, 2023, and submitted by:ADM ASSOCIATES, INC. NEEA staff is pleased to see the evaluator found NEEA's overall savings for the five-year reporting period to have a utility benefit cost-test (UCT) ratio of 5.68 for Idaho Power, 8.55 for Avista electric and 6.24 Avista gas. While we agree with this overall finding, we feel that it is important to acknowledge the methodology used in the evaluation is not consistent with how market transformation programs are evaluated.The evaluation used conventional impact protocols intended for assessing conventional resource acquisition programs. Evaluation of market transformation programs requires a different paradigm providing a comprehensive framework to assess the long-term strategy, program design, implementation and corresponding market response over the entire lifecycle of an efficiency innovation. NEEA uses the market transformation measurement and evaluation framework to guide independent,third-party evaluations of all our market transformation programs. Other regions, including California and Illinois, have also recognized this essential difference and have established frameworks specifically for this purpose as well. To make wise investment decisions, it is critical the appropriate measurement and evaluation protocols are utilized.This issue is discussed in more detail below. NEEA also has questions regarding several findings outlined in the report that do not appear consistent with the information we provided directly to the evaluator or the significant amount of publicly available documentation.The second part of this memo outlines these questions along with NEEA staff responses. With that said, NEEA staff are confident we can meet the general requirements of the seven recommendations articulated in your letter. However, Recommendation#9 will warrant further investigation and consulting with a NEEA Advisory Committee. Our responses to your questions are detailed in the table below. Northwest Energy Efficiency Alliance 700 NE Multnomah Street, Suite 1300, Portland, OR 97232 503.688.5400 1 Fax 503.688.5447 neea.org I info@neea.org Recommendation Can NEEA Impact/Timeline Comply? 1. Request NEEA to report annual Yes. Additional analytical labor costs associated with service savings for Idaho via service territory territory reporting. NEEA's current service territory methodology for each measure' approach will be used. Full implementation to start with claimed by NEEA. reporting 2025 energy savings. 2. Request that annual savings reports Yes. Additional labor costs associated with Idaho-specific include estimates of administrative reporting.This report will use the same service territory costs, incentive costs, and non- allocation used in reporting of savings.The report will incentive costs by service territory. provide an allocation of program administrative, incentive, and non-incentive costs. Develop report format in 2024 with implementation in 2025. 4. Track progress for each code change Yes. Additional labor costs associated with Idaho-specific relative to administrative dollars reporting. Develop report format in 2024 with spent towards state level codes and implementation in 2025. associated energy savings accrued by each state-level code. 5. Detail measure-level values as Yes. None; matches current practice. In the savings workbook accurately as possible, and that each provided annually to each funder, NEEA already enters all field is completed in the workbook to data fields for active programs and measures within each allow for year-over-year tracking of program for which we are reporting savings. NEEA regional units, baseline units, references all the data sources in the workbook. retirement units and unit energy savings over time. 6. NEEA to distribute naturally occurring Yes. None. Per our current practice, NEEA distributes naturally baseline units more equitably occurring baseline units equitably between local programs between local program units and and total regional units today, and no further adjustment total regional units. is recommended. NEEA staff would be happy to have a discussion to further clarify and understand the concern. 8. Complete third-party influence Yes. None.This is part of NEEA's current practice today. evaluations for all federal standards claimed by NEEA, as well as any future standards in which NEEA hopes to claim savings for the future. Using quantitative estimate of influence calculate a naturally occurring baseline for each standard. 9. Complete influence evaluations for Possibly, NEEA already conducts third-party evaluation of its energy each code update to estimate NEEA's pending codes work. Since the current influence approach's qualitative and quantitative influence further application to reported energy savings was recommended towards the code update. discussion by CEAC, any changes would need to be discussed by that with the Cost committee. Possible implementation in 2025 with cost Effectiveness implications to be determined. and Evaluation Advisory Committee (CEAC). Northwest Energy Efficiency Alliance -2 - Detailed NEEA Responses to Findings and Recommendations Following are NEEA questions and responses to the findings and recommendations of the Evaluation of NEEA Impacts Allocated to Idaho Power Company and Avista Utilities Within the State of Idaho report;April 6, 2023;ADM Associates San Ramos, California. The following section of this memo is organized to match "1.4 Findings and Recommendations" of the Evaluation, highlighting where a misunderstanding of the Market Transformation approach may lead to misleading or incorrect assumptions. General Comments Overall,the Evaluators found that that there were substantial energy savings resulting from the utilities investments in NEEA over the five-year period 2017-2021 and that in aggregate investment in NEEA over multiple years resulted in highly positive utility benefit-cost test ratio, shown in Tables 1-4, 1-5 and 1-6. Going deeper, the evaluation found that the contribution to NEEA efforts for standards, and codes was highly cost-effective across program years 2017 through 2021,with cost-benefit ratios ranging between 11.92 to 167.66, except for one code program in Avista Gas. However,the Evaluators found that all efficiency measure efforts were not cost effective for all program years,with cost-benefit ratios ranging between 0.0 and 0.7. The evaluation also looked at the methodology used to allocate and report energy savings specific to Idaho utilities participating in the alliance. Compared to an allocation methodology based on Idaho-specific service territory,the evaluation determined that using funding share allocation methodology, overestimated savings accrued out-of-state and underestimated savings accrued within Idaho.The Evaluators used the ratio of estimated savings using service territory allocation to funder share allocation to develop "realization rates"for individual measures. The evaluation found realization rates ranging from under 100%to over 100%. However,the overall effect of this change revealed the collective utility investment in NEEA remained cost-effective for Idaho Power electric, Avista electric, and Avista gas due to codes and standards savings. NEEA Response: NEEA staff are in general agreement with the overall finding that there were substantial energy savings resulting from the utilities' investments and participation in the alliance over the five-year period 2017-2021 and that, in aggregate, investment in NEEA over multiple years resulted in a highly positive utility benefit-cost test score, shown in Tables 1-4, 1-5 and 1-6. That said, NEEA notes that the methodology used in the report is inconsistent with the accepted practice of evaluation for long-term strategic market transformation programs. The report used an approach that isolates a single component of the market transformation implementation strategies deployed, or isolates a truncated time horizon, during the market transformation program life cycle.The full life cycle encompasses activities that range from emerging technology demonstration, market barrier removal, as well as activities to advance codes and standards. NEEA documents these activities in an overall market transformation logic model that is evaluated annually through market progress indicators. For any efficiency opportunity that has an applicable code or standard, these elements are targeted from the earliest strategy and implementation of the program.Conducting a cost-effectiveness analysis of a market transformation initiative for only a portion of the program's life, or at an individual measure level,will lead Northwest Energy Efficiency Alliance - 3 - to results that may drive incorrect decisions. See NEEA's response to Recommendation#2 for further explanation. For example, the alliance (NEEA staff and regional utilities and efficiency program administrators) has been advancing electric heat pump water heater technology through targeted interventions and incentive programs since 2008. From the very beginning of this work, NEEA developed a market transformation program that included elements of emerging technology development for cold-climates, retail and distributor engagement, contractor training, consumer marketing, ENERGY STAR®specification development, as well as deliberate collection and reporting of Northwest-specific cost and performance data into the federal standards process to ensure that Northwest regional needs are incorporated into a national Heat Pump Water Heather(HPWH) standard that is currently under consideration. During the early years of HPWH emerging technology development, a cost-benefit analysis for any one year would have looked infinitely non-cost-effective. Conversely, an annual view of the cost-benefit ratio following federal standard implementation will likely look highly cost-effective;yet the standard could not have been adopted without all the emerging technology and market development work that preceded it. To address this challenge, NEEA uses a 20-year net-present value cost-benefit analysis that accumulates costs and savings over the full span of market intervention activities that result in increases in market adoption that, in turn, result in measurable energy savings. We believe this is the appropriate way to represent the long-term nature of the benefits and costs of market transformation programs which seek to achieve the full potential of the efficiency innovations targeted. This is illustrated below in our response to Recommendation#2. The segregation and separate cost-benefit analysis of the integrated market transformation is a representation of a broader problem with the overall evaluation framework. The ADM team appears to have used a standard resource acquisition program evaluation framework. Market Transformation programs require a different framework for evaluation and energy savings assessment as they are fundamentally different in their approach, implementation, and long-term nature. NEEA uses an evaluation approach and savings assessment methodology that has been developed over the last 25 years, in collaboration with Northwest stakeholders,that include regulatory commissions of all four states.This comprehensive approach is consistent with other states that have engaged in market transformation programs including California'and Illinois.' NEEA staff also have questions regarding the approach and data ADM used to both determine realization rates, as well as their service territory allocation method. See NEEA's response to Finding#9 below for more information. Section 1.4.1. Specific Comments by Findings: • Finding#1: Utilities that fund NEEA can choose whether savings are reported by allocation share methodology or service territory methodology.The allocation shares methodology overrepresents 'Building a Policy Framework to Support Energy Efficiency Market Transformation in California, Prahl,and Keating, California Public Utilities Commission, October 2014. https://pda.energydataweb.com/#!/documents/1207/view '2022 Illinois Statewide Technical Reference Manual for Energy Efficiency Version 10.0 Volume 4:Cross-Cutting Measures and Attachments FINAL;Attachment C Framework for Counting Market Transformation Savings in Illinois, September 24, 2021 https://icc.illinois.gov/downloads/public/2022%201L- TRM%20Version%2010.0%20Volume%204%20Cross-Cutting%20Measures%20and%20Attachments%20(Final).pdf Northwest Energy Efficiency Alliance -4 - out-of-state and out-of-service territory savings across measures, codes, and standards while simultaneously underrepresenting in-state and in-service-territory savings across measures, codes, and standards. However,the service territory methodology accurately represents benefits directed to Avista and Idaho Power customers within the state of Idaho. NEEA Response: NEEA staff was unable to replicate all of the service territory allocations used to compute the realization rates that ADM reported, presumably from the data NEEA provided. Not all NEEA data has resolution that readily supports individual service territory reporting. Accordingly, NEEA staff apply specific methodologies to analyze and allocate data when approximating service territory level allocation. Our uncertainty of ADM's full understanding and application of the appropriate methodology, or data, leaves NEEA staff with significant concerns with this finding. We were not able to replicate the results reported by ADM using our own data and therefore would need more discussion with ADM to understand their analysis and data used before we could fully support this finding. Please see the NEEA response to Finding#9, regarding NEEAs concern that the evaluator utilized incomplete and unverified data. For these reasons, as well as the regional nature of the work and how technologies diffuse across markets, most alliance utilities have chosen to report savings by funder share. However, NEEA staff will continue to offer and provide either option upon request. • Finding#2:The data NEEA utilizes to estimate net market savings is available at resolutions that allow NEEA to estimate precise savings for each utility service territory. NEEA Response: It is incorrect to state that NEEA has data at resolutions to estimate precise savings by utility. We have data at varying levels of granularity for the programs in our portfolio, and this data can have levels of variability each year, as we rely on market relationships to obtain data and that can change over time for any given program. "Net market effects" are a calculated category of savings and are dependent on the total market adoption for a given year, a regionally forecasted estimate of the naturally occurring baseline, as well as the information provided by utilities for their local program, any of which can vary at any time, as far as granularity. • Finding#3:The Evaluators found that the methodology in which savings were estimated across measures were inconsistent. For some measures, service territory methodology was used, and for others,funder share allocation methodology was used. NEEA Response: As noted by NEEA staff and in the report, allocation method for reporting savings to Avista changed, per their decision, in 2019 from service territory methodology to funder share allocation methodology. NEEA would like to understand the specific areas of inconsistency ADM found. • Finding#4: NEEA prioritizes cost-effective savings in terms of regional benefit.Therefore, savings and cost-effectiveness are distributed across the region evenly, despite observed distribution of savings across states.Although this philosophy has merit, more precise estimates of utility-level and program-level savings help NEEA's stakeholders relay relevant savings and cost-effectiveness results to their respective regulatory commissions.This remains critical, due to some state-level commission orders to pursue all cost-effective energy efficiency efforts. Northwest Energy Efficiency Alliance - 5- NEEA Response:This is an incorrect finding as NEEA does not distribute cost effectiveness across the region, rather NEEA employs an assessment methodology at the regional level. NEEA does not then distribute the outputs of this assessment across the region. NEEA's practice is to work individually with each utility, per their request,to help relay savings and cost-effectiveness results to their respective regulatory commissions. • Finding#5:The interviews revealed that although the three parties fundamentally want to improve energy efficiency and increase market adoption of emerging technologies,their preferred approaches to this shared goal vary. Unlike the utilities, who strive to demonstrate the cost- effectiveness of their initiatives and investments on an annual or bi-annual cycle, NEEA operates on a five-year funding cycle, which is different than the typical annual or biannual utility planning cycle. NEEA Response: It is correct that NEEA operates on a five-year funding cycle. This funding structure is designed intentionally to recognize the long-term nature of market transformation and allow for consistency and continuity in the investments required to develop and execute on such programs in our portfolio. Consistent with NEEA's regional portfolio management approach and the long-term nature of market transformation, NEEA regularly reviews the benefits and costs of its programs and portfolio to ensure a benefit-cost ratio greater than 1.The benefit-cost ratio represents a long-term forecast across the full development lifecycle of transformation. The assumptions used in the forecast for each market transformation program are reviewed annually and relevant updates made as needed; this updated result is then reported annually. Given the difference pointed out in the recommendation, we are unclear how ADM's finding is reflected in the methodology used in this report and in the recommendations. • Finding#6: NEEA's programs are designed with a broader constituency in mind than that of its member utilities. While the Idaho utilities' programs are targeted to produce benefits for their ratepayers,—NEEA is tasked with developing programs that need to consider what is best for the entire four-state region.At its core, NEEA's ethos assumes that changes made in one state will eventually spillover into another state and that eventually, regional change will be realized. NEEA Response: NEEA agrees in general with this finding. However,we disagree with the final statement of an assumption that changes in one state will "eventually spillover" into another state. NEEA's purpose is to pool resources and share risks to transform the market for energy efficiency to the benefit of all consumers in the Northwest, including customers of Idaho utilities.To accomplish this goal, NEEA focuses its efforts on engaging decision-makers (e.g., manufacturers, retailers, and distributors) in competitive markets that often cross utility service territories and state boundaries. As a result, NEEA's market transformation work is designed to work and result in changes that affect all four states in the region. Sometimes this happens in all four states simultaneously(e.g., federal appliance standards) and in other cases,the transformation happens over time across the region. Regardless, NEEA works to ensure that market actors are engaged in all four states and that the entire four-state region is transformed.There are examples (e.g., Building Operator Certification, Heat Pump Water Heaters) where NEEA has continued to fund transformation activities in Idaho and Montana to ensure that transformation that started in Washington and Oregon continues in the eastern part of the region. This active engagement is a commitment to ensuring that NEEA's purpose is fulfilled to all utilities and fundamentally differs from the assumption of the spillover statement in this finding. Northwest Energy Efficiency Alliance -6 - • Finding#7: NEEA currently allocates code savings via funder share methodology, which estimates a proportion of total NEEA funding to each utility based on number of electric retail customers and overall load.Therefore, savings from code adoption in other states are in-part assigned to Idaho. The Evaluators found that out-of-state code building savings are currently being attributed to Idaho utilities.The Evaluators are skeptical that spillover from out-of-state code changes result in energy savings within the state of Idaho.Although the barriers to code adoption from one state to the next may be similar,there is no evidence to suggest that these learnings transfer to observable and measurable savings. NEEA has stated that starting in 2022, code savings will be allocated via service territory allocation. NEEA Response: NEEA would like to understand this finding and its basis, especially the statement by ADM that they are skeptical about spillover. NEEA has never claimed that building savings from a code change would spill over into another state. Rather, until recently, code adoption in one state influenced code adoption in an adjacent state, and/or the International Energy Conservation Code, which is the basis for many state energy codes. Using the funder share allocation method, we saw savings being allocated proportionate to the individual state's code over time. Prior to this evaluation, NEEA decided to change its allocation method for codes, due to the significant changes we were seeing across the states relative to policy and associated impact on codes timing and stringency. NEEA no longer allocates energy savings from codes using a funder share allocation method, and instead will use a state allocation methodology. • Finding#8:The NEEA Cost Effectiveness Advisory Committee (CEAC) meets quarterly with the NEEA objectives to provide space for discussion around results of recently completed evaluation, progress of field studies, relevant updates to programs, and acceptance or questioning of NEEA methodology towards calculation of savings. NEEA Response: CEAC are public meetings, and NEEA encourages all stakeholders and interested parties to attend. Efficiency Measure Findings • Finding#9:The Evaluators estimated verified Ex Post aMW for the efficiency measures to display 39%, 52%, and 0% realization rates for Idaho Power electric,Avista electric, and Avista gas savings within the state of Idaho, respectively.The difference in claimed savings and verified savings is due to the change to using service territory allocation rather than funder share allocation.The efficiency measures category Ex Ante savings included savings for measures completed in Washington, Oregon, and Montana—therefore,for some measures,the funder share allocation methodology underestimated Idaho-specific savings while others overestimated out-of-state savings.The overall effect of this change resulted in a lower than 100% realization rate. NEEA Response: It appears that ADM used preliminary, incomplete, and unverified data in their attempt to generate a service territory view of NEEA savings. NEEA has a standard practice of ensuring that all data is verified and complete before making a final assessment and reporting of savings by the allocation selected by the utility. To be responsive to ADMs request in early January, NEEA staff provided access to data that was preliminary, incomplete, and unverified. Staff informed ADM of this fact and asked that the data not be used because it was not fully updated or verified and was not intended to be used in any reporting to the utilities. Northwest Energy Efficiency Alliance - 7 - Additionally, as stated in the NEEA Response for Finding#1, NEEA has concerns that ADM did not properly employ the correct methodology or analysis required to estimate service territory level savings. • Finding#10:The database review revealed that a variety of fields (measure life, UES)were empty across measure types due to lack of savings claimed for the measure,which made verification of values difficult and complicates tracking of a measure progress overtime. NEEA Response:ADM did not have access to any NEEA database; however,they did have access to the data sets that populated savings reports for the funders and years requested. These data sets come from a database that includes measures for which NEEA did not report energy savings for these funders, and hence those cells were empty. • Finding#11:The database review revealed that NEEA's current method for distribution of modeled naturally occurring baseline units between local program and NEEA efforts is not reasonable.A portion of energy-efficient technology sales are due to naturally occurring baseline. NEEA nets out modeled naturally occurring baseline to avoid claiming savings for units that would have been sold had no program or NEEA-effort been provided within the market. However,the method in which these baseline units are netted out is not distributed equitably. For some measures, NEEA estimates that a large proportion of local program units are baseline, and therefore a larger proportion of the remaining net market effects is assigned to NEEA efforts.The Evaluators raise concern for this assumption, as it is unlikely locally incentivized, rebated measures display the same free ridership as non-incentivized measures in the region. NEEA Response: As stated in response to Finding#10, it is important to clarify that ADM did not have access to any NEEA database. NEEA disagrees with ADM's assertion that the distribution method of baseline units NEEA uses is not reasonable or equitable. A foundational principle of NEEA, as an alliance, is that local programs support market adoption and are therefore a part of the market transformation taking place. NEEA's baseline market share estimate represents the adoption that would have occurred without the intervention of NEEA or its funders' local programs, so NEEA's approach assumes that a portion of the baseline market share applies to the local incentive units. Regarding the question of equitability, NEEA does not assign savings to NEEA efforts separately from funder programs; rather, it measures the full market transformation savings from our collective efforts (Co-created Savings) and reports that to funders.To help our funders avoid double-counting, in our savings reports we net out local program savings.The remaining savings, called net market effects savings are not representative of distinct NEEA efforts, nor do they reflect attribution. • Finding#12:The Evaluators reviewed the utilized UES via the Regional Technical Forum (RTF) workbooks,field study data, and simulation analysis findings and note no large concerns with NEEA UES methodology or market baseline assumptions. NEEA Response: It is NEEA's practice to align with the RTF, unless we have newer data. NEEA works closely with RTF and Council members to discuss any new information or methodology that is pertinent to analyses done by both NEEA and the Council. Northwest Energy Efficiency Alliance - 8 - • Finding#13:The Evaluators found that NEEA calculates cost-effectiveness of its portfolio using the total regional savings rather than the net market effects.The Evaluators determined that this methodology raises concern, and the NEEA cost-effectiveness tests currently account for all measure, standard, and code completions across the entire region, effectively double counting local program savings and simultaneously claiming naturally occurring baseline savings. Because Avista and Idaho Power calculate their own internal cost effectiveness tests, this finding does not impact Idaho Power or Avista reporting. However, the Evaluators highlight this finding, as NEEA savings allocation and cost allocation methods are not currently consistent with regulatory requirements. NEEA Response: NEEA uses a Total Resource Cost (TRC) test which uses total regional costs and total regional savings to assess the cost-effectiveness of both individual market transformation programs as well as the portfolio as a whole.The TRC methodology is consistent with that used by the Northwest Power and Conservation Council'and the principles embedded in the Northwest Power Act.4 This approach looks at cost-effectiveness from a societal perspective inclusive of all costs, including NEEA, consumer and local utility program administration costs, and benefits over a long-term, 20-year planning horizon. As a regional organization, NEEA has used TRC as an appropriate tool for making decisions regarding the pursuit of new market transformation opportunities as well as managing the overall portfolio. The TRC methodology and application to individual programs is reviewed by both the CEAC as well as by third-party evaluation contractors who independently review each program. NEEA staff refutes the statement in the finding regarding double counting of local program savings. In fact, NEEA collects annual data on energy savings units claimed through local utility programs and subtracts them from total market adoption units to avoid double counting of energy savings reported to utilities and regulators. NEEA also deducts savings associated with the naturally occurring baseline units. It is also incorrect that NEEA includes all standard and code completions across the region in the cost effectiveness assessment of the portfolio. Each program is assessed separately and may include a code or standard associated with the program strategy. However not all savings recognized in code or standard results are included in the portfolio cost effectiveness assessment. NEEA recognizes that units adopted into code will have a different, quite often lower, unit energy savings, since the baseline replacement unit in new construction code is typically higher performing than a baseline replacement unit in retrofit practice, and so this higher baseline condition is netted out. Additionally,to be conservative in the portfolio-level benefit cost calculation, NEEA includes all of the regional investment dollars in NEEA, including early lifecycle costs invested in programs, new construction codes and federal and state standards work in development, regional building stock assessments and all general administration,while only recognizing the benefits of a portion of the programs in our portfolio that are currently funded, fully developed,vetted and in the market development phase of their lifecycle. 'Cost-Effective Methodology(nwcouncil.org) 4 Northwest Power Act(USC numbered)(nwcouncil.org) Northwest Energy Efficiency Alliance - 9 - Given this, NEEA is unclear about the specific concern mentioned by ADM. NEEA has always been, and remains, committed to providing necessary data to all NEEA funders in their individual regulatory filings. NEEA recognizes that Idaho utilities are subject to a Utility Cost Test metric that is distinct from the TRC metric that NEEA uses. NEEA is happy to address any concerns or clarifications needed to support Idaho Power and Avista in their regulatory filings in Idaho. Standards Findings • Finding#14:The Evaluators estimated verified Ex Post aMW for the standards efforts to display 34% and 50% realization rates for Idaho Power electric and Avista electric within the state of Idaho, respectively. Avista gas did not claim any savings for standards.The difference between claimed savings and verified savings is due to the change to using service territory allocation rather than funder share allocation. A minor cause of discrepancy is due to corrected baseline units using influence evaluation values. NEEA Response: Please see response to Finding#9, as it applies to this finding as well. In addition, it appears that savings from one standard,which was previously evaluated by a third-party evaluator for influence and impacts across the states within the region,were removed. All standards in which NEEA is reporting co-created savings have been evaluated and are available. • Finding#15: NEEA contracts third-party evaluators to conduct "influence evaluations"for each standard, which summarizes NEEA's overall qualitative and quantitative influence towards federal standards updates. NEEA uses the quantitative assessment as an estimate of federal standards naturally occurring baseline.The Evaluators found that some of these influence scores were not integrated properly to estimate baseline units.The Evaluators also found more than half(13 of 25) federal standard measures lack influence evaluations. NEEA Response: NEEA does not agree with these two assertions:that influence scores were not integrated properly and that reported energy savings lacked evaluations.The variance was either due to rounding or a reduction of the percentage based on NEEA's participation (Fluorescent Lamp Ballasts). There is one case (Pumps)where the assumptions contained a preliminary value and is to be updated based on the final evaluation value. In cases where NEEA reported co-created energy savings and ADM is stating no evaluation was conducted, they are mistaken. NEEA reviewed all appliance standards for which we claimed co-created savings and confirmed evaluations are available in each instance. Lastly, it appears that the evaluator did not include any energy savings associated with a state appliance standard that NEEA influenced named "Residential/Commercial Battery Chargers Standards effective in Oregon in 2014.They claimed there was no evaluation conducted. A third- party evaluation was conducted, and results reviewed by the Cost Effectiveness and Advisory Committee in 2015.The report found that NEEA did influence the standard, and the majority of battery chargers sold in Idaho, Oregon and Washington met this new standard. A copy of the evaluation is posted on NEEA's website and can be found at https://neea.org/resources/2015- battery-charger-standard-evaluation-for-neeas-non-adoptive-states. Northwest Energy Efficiency Alliance - 10- Code Findings • Finding#16:The Evaluators estimated verified Ex Post aMW for the code efforts to display 137%, 125%, and 87% realization rates for Idaho Power electric,Avista electric, and Avista gas savings within the state of Idaho, respectively.The difference between claimed savings and verified savings is due to the change to using service territory allocation rather than funder share allocation. Overall,the funder share allocation underestimated Idaho-specific code savings using the current NEEA policy of claiming 100%code after code is implemented. NEEA Response: NEEA staff was unable to replicate the realization rates ADM computed from the data NEEA provided. NEEA staff agree that State Energy codes were understated during the reporting time frame of this evaluation when using the funder share allocation methodology. • Finding#17: Currently, NEEA does not complete third-party evaluations of NEEA"influence" towards codes updates as is currently done for federal standards updates.Therefore, NEEA currently claims 100%savings for code-built homes.As summarized in the standards influence evaluations summarized in Table 3-35 of the report,NEEA influence towards standards ranges between 2.6%and 61%. If codes are evaluated similarly, and portray a similar range of influence, NEEA code savings could be significantly overrepresenting savings. NEEA's current policy is to report 100%of code-built residential and commercial building savings (while integrating compliance rates)for 10 years after the effective code update date. Currently, NEEA does not maintain a model to estimate naturally occurring baseline over time, as it does for its energy efficiency measures.The current NEEA methodology assumes that there would be a 10-year lag in current residential and commercial building code if NEEA did not participate in code update efforts. NEEA Response: NEEA would like to understand the basis for this finding. Specifically,what rationale ADM found that state energy building codes should be treated the same as Federal Appliance Standards. NEEA has provided a significant portion of the overall code support funding in all four states over the last 25 years of NEEA's existence. NEEA's funding has also provided representation for the four states in the model code development process (IECC) and affiliated buildings standards processes (DOE,ASHRAE,AHRI). NEEA also conducts code compliance and new construction practice assessments for each state,this provides the foundation for future code development within the region and nationally. Given the continuous and substantive nature of NEEA's work in the state and national model building codes,the CEAC developed and recommended the use of a standardized influence factor of 10 years, essentially three normal code- cycles, of acceleration of code provisions that NEEA has worked directly on. In addition, NEEA evaluates code compliance in each of the four states and applies this to the savings it reports. Finding#18:The Evaluators reviewed simulation model methodology used by NEEA to estimate code savings and found that UES methodology for code savings do not present any concerns. NEEA Response: No response. Northwest Energy Efficiency Alliance - 11 - 1.4.2 Recommendations. Specific Comments by Recommendation. • Recommendation#1:The Evaluators recommend Avista and Idaho Power request NEEA to report annual savings via the service territory methodology for each measure claimed by NEEA for each Idaho Power electric,Avista electric, and Avista gas. (Based on Finding#1,#2,#3) NEEA Response: NEEA can report annual savings and an estimate of costs to each utility using NEEA's current service territory approach. NEEA seeks to have data for each of its market transformation initiatives (MTI)to achieve this level of reporting.There are times, where either the market data is not available or cost prohibitive at that level of granularity, and in these circumstances, NEEA will report at the level it has the best data available. On page 53, ADM agrees with this by caveating—"if source data permits."Annual reports containing energy savings and estimated costs are available in April,for the prior year, and forecasting at a service territory level is not possible. NEEA has already committed to Idaho Power staff,to discuss the level of detail available in a given year so that the content and final reported values for this methodology are understood.There is additional NEEA analytical and administrative labor costs associated with this recommendation. • Recommendation#2:The Evaluators recommend that Avista and Idaho Power request annual savings reports to include estimates of administrative costs, incentive costs, and non-incentive costs by service territory.This will allow each utility to calculate more accurate cost-effectiveness tests for each initiative to determine whether extension of funding is a viable option within each utility's regulatory environment. (Based on Finding#4) NEEA Response: See NEEA's response to Recommendation#1 regarding inclusion of cost data. NEEA does not agree that calculating cost effectiveness on an annual basis for a long-term multi- phase market transformation initiative (MTI) is accurate. Doing so could hinder advancement of any emerging efficient technology and,therefore, not be used to influence a code or standards in later years, once the market barriers were removed and market adoption was occurring. Critical to managing MTIs is acknowledging the long-term horizon (10-20 years) of the program investment and return on investment in the market. A resource acquisition (RA) approach, which is a one-to-one transactional approach over a short time horizon (1-3 years), is limited in its value beyond the immediate efficiency gain the purchase creates. A typical cost-effectiveness construct for a RA program is a 1-to 2-year comparison of dollars in vs energy savings produced. In contrast, an MTI is designed and implemented to make sustained changes in markets that ultimately result in energy savings. For this reason,the MTI cost and benefit period should span a 20-year horizon.This covers the full-term of market investment, response, and value realized (similar to an R&D model). The early years of a MTI are high investment years requiring resources to identify, address, and remove barriers in the market so that the system provides the energy efficiency options to the consumer. In later years,when the barriers are reduced/removed and the MTI investment can cease,the market has established consistent practices, the efficient option becomes widely available in the supply chain and known to the consumer, and the data and market information can be used to influence a change in an energy code or appliance standard, and the full value of the investment is realized.The figure below illustrates the diffusion of innovation and how an MTI investment inversely relates to the value delivered, in terms of sustained market adoption and energy efficiency. Northwest Energy Efficiency Alliance - 12- Choosing a narrow time slice of an MTI for assessing cost-effectiveness will yield misleading results as it will not capture the full scope of the MTI investment and value return phases. It will also not support proper decision-making. • Recommendation#3:The Evaluators recommend that NEEA work with utilities to accurately produce service territory-level savings and to best serve each state's current regulatory environment and utility's localized concerns. (Based on Finding#5) NEEA Response: NEEA works closely with its funders to understand their concerns and their regulatory environment. NEEA is happy to work with Idaho Power Company and Avista Idaho to produce service territory reporting. • Recommendation#4:The Evaluators recommend that NEEA track progress for each code change relative to administrative dollars spent towards state-level codes and associated energy savings accrued by each state-level code. With the 20-year market transformation in mind,the service- territory-level savings will still accrue over the 20-year horizon, however, using this methodology, actual market transformation effects of co-created savings will be more accurately tracked. (Based on Finding#6,#7) NEEA Response: NEEA can provide this information.There is additional NEEA administrative labor costs associated with this recommendation. • Recommendation#5:The Evaluators recommend that measure-level values are detailed as accurately as possible, and that each field is completed in the workbook to allow for year-over-year tracking of regional units, baseline units, retirement units, and unit energy savings values over time. (Based on Finding#10) NEEA Response: NEEA provides all detailed data annually as called out in recommendation#5. The only missing data, as finding#10 states,were fields intentionally left empty due to NEEA not reporting savings for that measure. Recommendation#6 The Evaluators recommend that NEEA distribute naturally occurring baseline units more equitable between local program units and total regional units. (Based on Finding#11) NEEA Response: NEEA staff use a well-vetted, equitable approach to allocating baseline unit for all funder reporting calculations, applying a weighted average across the region based on total regional results and local unit acquisition by each funder in the alliance. This approach has been reviewed extensively by CEAC. NEEA staff is unable to discern from the report exactly what elements of this approach ADM finds inequitable. NEEA staff would be happy to discuss this further ADM or Idaho power and Avista staff as appropriate. • Recommendation#7: In the case that cost effectiveness tests are completed using NEEA- reported savings,the Evaluators recommend that Avista and Idaho Power calculate cost-effectiveness using net market effects rather than total regional savings,as is consistent with current regulatory requirements to report gross savings that would not have occurred without program intervention. (Based on Finding#13) Northwest Energy Efficiency Alliance - 13- NEEA Response: Assuming the cost effectiveness test referenced in this recommendation is a utility cost test, NEEA recommends co-created savings be used in the analysis.As shared with ADM, NEEA employs an accounting approach to segmenting energy savings for reporting. Co-created savings represent what was achieved by both the local program effort and the regional market transformation effort. "Net market effects" is a calculated category of savings, and it is the savings that are remaining after both baseline and local program units are removed. The sole purpose of this category is to not double count savings when reporting; it is not intended to assign attribution. Additionally, NEEA recommends using a window of time that is representative of the full cycle of costs and benefits, as benefits lag the cost outlay, and a narrowly defined time horizon can yield misleading results. Please see NEEA's response to the Overall Finding and recommendation#2 for more information. • Recommendation#8:The Evaluators recommend that third-party evaluations are completed for the federal standards claimed by NEEA, as well as any federal standards in which NEEA hopes to claim savings for in the future. Using the quantitative estimate of NEEA influence,the Evaluators recommend that NEEA calculate a naturally occurring baseline for each standard. (Based on Finding #15) NEEA Response: NEEA has been operating in full compliance of this recommendation. NEEA conducts third-party evaluations of all federal standards in which NEEA sought to influence.These evaluations are conducted after the standard has been adopted, and prior to reporting any energy savings. • Recommendation#9:The Evaluators recommend an evaluation is completed for each code update to estimate NEEA's qualitative and quantitative influence towards the code update. (Based on Finding#17) NEEA Response:As outlined in finding# 17, NEEA's significant influence in the state building energy code process is distinct from federal appliance standards and therefore we would like to understand ADM's basis for this recommendation before suggesting any changes to process or reporting. If there were a change, their most likely would be cost implications to NEEA. Thank you again for the opportunity to respond. Northwest Energy Efficiency Alliance - 14-