The URL can be used to link to this page

Home My WebLink About20040519Responses 207 226 244 of Avista to Staff.pdfAvista Corp. 1411 East Mission PO Box 3727 Spokane, Washington 99220-3727 Telephone 509-489-0500 Toll Free 800-727-9170 ~~' 'V'STAOO Corp. May 18 , 2004 Idaho Public Utilities Commission 472 W. Washington St. Boise, ill 83720-0074 1\ttn: Scott Woodbury Deputy 1\ttomey General Idaho Public Utilities Commission Office of the SecretaryRECEIVED MAY 1 9 2004 Boise. Idaho Re:Production Request of the Commission Staff in Case Nos. 1\ VU-04-01 and 1\ VU-04- Mr. Woodbury, I have attached an original and three copies of 1\vista s response to Staff Data Request No. (s) 207, 226, and 244. If you have any questions, please call me at (509) 495-4706. ~elY' Mike Fi Rate i\nal yst Enclosures Copy: C. Ward (Potlatch) D. Peseau (Utility Resources, Inc) A. Yankel (Yankel & Assoc., Inc) VISTA CORPORATION RESPONSE TO REQUEST FOR INFORMATION JURISDICTION: C1\SE NO: REQUESTER: TYPE: REQUEST NO. Idaho A VU - E-O4-0 1 / A VU -O4-0 1 IPUC Data Request Staff - 207 D1\TE PREPi\RED: WITNESS: . RESPONDER: DEP AATMENT: TELEPHONE: 5/17/2004 Bob Lafferty Jon Powell Energy Resources (509) 495-4460 REQUEST: Beginning on page 13 of his direct testimony, Robert Lafferty described an independent review of 1\vista s analyses of supply-side resources. Was a similar independent review of 1\vista analyses of demand-side resources conducted? If so, please provide the review. Ifnot, please explain why such a review was done for supply-side but not for demand-side. RESPONSE: i\n independent review of analyses of demand-side resource proposals was not conducted given the relative scope and nature of the proposed projects coupled with the fact that a large staff group critically reviewed the analyses of the demand-side resource proposals. VISTA CORPORATION RESPONSE TO REQUEST FOR INFORMATION JURISDICTION: CASE NO: REQUESTER: TYPE: REQUEST NO. Idaho AVU-O4-01 / AVU-O4- IPUC Data Request Staff 244 D1\TE PREP1\RED: WITNESS: RESPONDER: DEP AATMENT: TELEPHONE: 05/10/2004 Bob Lafferty Rick Lloyd Resource 1\ccounting (509) 495-4935 REQUEST: Please show how maintenance and management costs for Coyote Springs II are split with Mirant. Also, show how the costs are booked, by FERC account. Provide the costs both on a system level and at the Idaho jurisdictional level. RESPONSE: 1\vista and Mirant are signors to the "Co-Tenancy and Joint Operating 1\greement". Various appendixes to the agreement spell out responsibilities for maintenance, management and capital costs. 1\ppendix G - Section 8 states: Variable O&M Expenses (other than L TS1\ Costs) will be allocated monthly based on the percent of generation taken during the month the expenses were incurred and Fixed O&M Expenses will be allocated based on each Owner s Pro Rata share of the Facility. The "Long Term Service Agreement" costs (LTS1\ costs) with General Electric are allocated based on Avista s and Mirant's run times by month, and plant starts and stops initiated by each party. 1\vista s share of operating, maintenance and capital costs are recorded monthly by journal entry. 1\ summary of charges by FERC functional account for 2003 (7/1/03-12/31/03, excluding fuel) is attached. Sum of TRAN AMT JID U MAIN SUB Total 5001 0 39. 50151 96.81 54600 528.42 551 00 243. Total 907. 54751 744. 55300 943. Total 687. 54751 367. 55300 448. Total 815. 54751 535. 55300 169. Total 705. 54751 222.40 55300 Total 222.40 54751 769. 55300 89. Total 858. 54600 256 990. 54720 15,479 395.47 54800 196 150. 54900 207. 55000 755. 55300 974 018. 55400 168. 55716 205. 56200 070. Total 075 960. 55300 33. Total 33. Grand Total 099 192. fuel fuel fuel fuel fuel fuel Less Fuel 547 Total O&M (15,495 034.97) 604 157. Staff Data Request No. 244-Attachment VISTA CORPORATION RESPONSE TO REQUEST FOR INFORMATION JURISDICTION: CASE NO: REQUESTER: TYPE: REQUEST NO. Idaho A VU-O4-01 / A VU-O4- IPUC Data Request Staff 226 D1\TE PREPi\RED: WITNESS: RESPOND ER: D EP 1\R TMENT: TELEPHONE: 05/17/2004 Bob Lafferty J on Powell Utility Finance (509) 495-4047 REQUEST: Please provide any available analyses of the cost effectiveness of NEE A's programs for 1\vista Idaho customers. RESPONSE: The Northwest Energy Efficiency 1\lliance (NEEA) recently completed a "Retrospective Report complete with independent oversight. The focus of the study was to review of the cost- effectiveness and energy acquisition ofNEE1\ ventures. 1\dditionally NEE1\ has completed a review of recent program activity in Avista s Idaho service territory. This report provides additional detail on the local delivery of NEE A's regional market transformation ventures. These two documents are attached for your review. SUMMIT BLUE N S U LT l G, L L C FINDINGS AND REPORT RETROSPECTIVE ASSESSMENT OF THE NORTHWEST ENERGY EFFICIENCY ALLIANCE Final Report Prepared for: Northwest Energy Efficiency Alliance Ad Hoc Retrospective Committee Chair Mark Kendall, Oregon Office of Energy Ken Keating, Bonneville Power Administration Darlene Nemnich, Idaho Power John Savage, Oregon Public Utility Commission Ken Canon, Industrial Customers of NW Utilities Portland, Oregon December 8, 2003 Daniel M. Violette , Phd Michael Ozog, Phd Summit Blue ConsultirYcl Boulder, Colorado Phone 720-564-1130 dviolette~summitblue .com Kevin Cooney, MS , PE Stratus Consulting Inc. Boulder, Colorado Phone: 303-381-8000 kcooney~stratusconsu Iti ng. com CONTENTS Executive Summary...... ............................................................................................. ............ E- 2.4 Introduction 1.4 Background on Alliance History and Goals ................................................... Objectives of the Assignment .......................................................................... Project Activities ................................................................................................1-4 Layout of the Report ......................................................................................... Framework and Approach Assessing Market Transformation (MT) Projects ......................................... Initial Interviews with Alliance Staff ................................................................ Approach to Program Analyses....................................... ........ ........... ............ 2- Alternative Hypotheses Elicited and Explored ..............................................2-4 2.4.Potential Alternative Hypotheses Explored during Interviews ........ Approach to Scenario Analysis Illustrated Example ................................. Definitions Used in Program Analysis ............................................................ Analysis of Energy Star Residential Lighting ntroduction to Program.. .................... ........................................ ........ ..... ......... Assessing Program Accomplishments .......................................................... Pivot Assumptions ................................................................................. Alternative Hypotheses.................... ...... ................... ............... ............ Assessment Findings........... ........ ~............ ................................................... ..... 3- Cumulative Savings .............................................................................. Levelized Cost ....................................................................................... 3 Cost-Effectiveness.................................. ...................... ........................ 3.4 Qualitative Assessment............................ ..... ............. ................ ...... .. Analysis of Energy Star Residential Windows Introduction to Program ...................................... .............................................. Assessing Program Accompishments .......................................................... Pivot Assumptions ............................................................................... Alternative Hypotheses ........................................................................ Assessment Findings ............................ ......... ................................................. .. 4-4 Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report Cumulative Savings .............................................................................. Levelized Cost ......................................... .............. ......... ....... ...... ....... ... 3 Cost-Effectiveness.................. ...... .... ....... ........ ........... ..... ..... ........... .... . 3.4 Qualitative Findings.. ...................................... .............................. ....... . Analysis of Build ing Operator Certification Introduction to Program .................................................................................... Assessing Program Accomplishments .......................................................... Pivot Assumptions ................................................................................. Alternative Hypotheses..... ........... ............... ......... ................ ................ 5- Assessment Findings............. ...... ......... ....................... ................................ ..... 5-4 Cumulative Savings .............................................................................. Levelized Cost............................... ............................. ................... ........ 3 Cost-Effectiveness..... ..................................... ........... ...................... ..... 5-3.4 Qualitative Findings .... ....................................................... .... ..... .......... Analysis of MagnaDrive Introduction to Program... .......................... ........................................ ............... Assessing Program Accomplishments .......................................................... Pivot Assumptions ................................................................................. Alternative Hypotheses ........................................................................ Assessment of Findings ...................................................................................6-4 Cumulative Savings ..............................................................................6-4 Levelized Cost .......................................................................................6-4 3 Cost-Effectiveness .................................. .............................................. 6- Qualitative Findings .............................................................................. Summary Analysis of Other Alliance Programs Program Selection........................................................ ............... ...................... 7- Overview of Successes and Lessons ............................................................ 7- Upstream Programs.... ................................... ....... .................... ............ Training Programs ................................... .............................................. 7- Entrepreneurial Programs ....................................... ................. ............ 7- 2.4 Consumer (End-use) Programs .......................................................... 7- Overall Value Assessment Estimated Impacts of the Alliance - Energy Savings and Levelized Costs ........................................................ ........................... ....... Overall Value of the Alliance.......... ........................... ................ ...................... 8- Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report Issues, Findings, and Recommendations Over-Arching Themes from this Assessment ............................................... Issues Related to Cost-Effectiveness and Evaluation Processes .............9-4 1 The Alliance Cost Effectiveness Model............................................. 9-4. Market Progress Evaluation Reports ................................................. 3 The Market Activities Report ............................................................... 9- 9.4 Recommendations .............. .......... ...................... ................... .............. ............. 9- Final Comments ....... ...................... ........ .............. .... ..................... ..... ........... .. Appendices Individuals Interviewed, Interview Guides Pivot Assumption and Output Distribution for Scenario Analysis Document Log Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report 111 EXECUTIVE SUMMARY This report presents the results of the independent evaluation of market transfonnation accomplishments through Northwest Energy Efficiency Alliance (Alliance) efforts since 1997. This retrospective evaluation effort was initiated by an ad hoc committee appointed by the Alliance board of directors 1 for the primary purpose of detennining whether the Alliance has transfonned enough markets to justify the costs of the Alliance. This key question was investigated by looking at Alliance activities, and the role the Alliance has played in markets where the Alliance has spent significant resources over the past several years. This assessment of Alliance activities also takes into account the work of other organizations and their impact on energy efficiency markets to avoid double counting market impacts and effects. This evaluation was initiated in April 2003 , and the work effort spanned approximately four months. 1 Over-Arching Themes from this Assessment 2 Estimated Impacts of the Alliance - Energy Savings and Levelized Costs 3 Overall Value of the Alliance E.4 Alliance Analysis Process and Issues Discussion 5 Recommendations 6 Final Comments The evaluation team employed an interactive process with the ad hoc committee chainnan and the executive director of the Alliance, as well as meetings with the committee at key project junctures to ensure that the evaluation was proceeding in a manner consistent with committee objectives. The team used the extensive body of material already available on Alliance project activities, including market assessments that were conducted on Alliance projects, market progress evaluation reports, computer models, and planning documents. Primary data collection activities focused on interviews with staff, evaluators, implementation contractors, market actors and others to better understand program mechanisms and elicit alternative hypotheses regarding market changes. 1 Over-Arching Themes from this Assessment The study team reviewed the infonnation ftom the assessment and developed six over-arching themes that represent the high level findings. Each of these themes is addressed in greater detail in the concluding chapter of the main report. 1 The ad hoc committee consisted of both Alliance board members and non-board representatives, as listed on thecover of this report. Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report THEME 1 The Alliance business culture is characterized by open communications, a focus on the planning and delivery of programs, and no discernable bias. The Alliance has developed a culture of adaptive management and continuous learning that has been communicated throughout the organization. Alliance personnel were open and direct in its communications with the study team. The project interviews with evaluators and implementers indicated that the Alliance worked with them to reach appropriate answers, and to continue to make the Alliance a "learning" organization. THEME 2 The Alliance has been successful at transforming, or contributing to the transformation of markets. The study team reviewed Alliance program evaluations, and interviewed researchers who conducted program specific evaluations, as well as other regional stakeholders and market actors involved in energy efficient markets in the northwest. Based on the information gathered, the study team determined that the Alliance made substantive contributions to transforming regional markets for energy efficiency equipment and practices. Specifically, the MPERs addressing the Energy Star Windows program were compelling in their documentation of Alliance market influence in terms of the increase in the number of active manufacturers of high efficiency windows in the region. Both the MPERs and national data indicate market penetration of Energy Star windows in the Northwest are more than twice the national average. Interviews with industry experts also supported the contention that Alliance activities ha~ permanently impacted the windows market. Other markets where there was considerable evidence of market transformation included clothes washers and CFLs. For clothes washers, the market penetration of Energy Star washers in the NW are significantly higher the national average (see chapter 7). The market for CFLs changed more dramatically in the NW than other regions of the country, even when the West Coast energy crisis is considered (see chapter 3). The magnitude of the impact of the Alliance on these markets was a subject that not all market actors in the region agreed upon, but there was a more general agreement that permanent changes had taken place in these markets and were at least partially the result of Alliance activities. THEME 3 Market Progress Evaluation Reports tended to focus on the program delivery process and on providing feedback for program design and implementation improvements. This met an immediate need for Alliance personnel responsible for program implementation, and these reports improved the delivery and implementation process. This trend towards process analyses and providing feedback for program improvement fits well with the Alliance s goal of adaptive management, but the estimates of impacts and the ability to substantiate claims of Alliance- induced market effects would have been enhanced by having the MPERs more directly address savings per unit and the issue of attribution within each study. In general, the MPERs were not structured to provide information on attribution or savings per unit for energy efficiency equipment (or applications) influenced by Alliance activities. THEME 4 : Cost-effectiveness analyses were difficult to replicate and the current processes used are cumbersome. Cost-effectiveness models and analysis efforts form the basis for proj ecting market impacts from programs under development by the Alliance and they are used as the basis for the Alliance claims of cost effectiveness as presented to the public in the MAR. Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report The initial pressures on the Alliaoce have been to develop and implement programs in the field. The study team commends this past focus, but is emphasizing the need for improvement in this area going forward. The magnitude of claims made by the Alliance as its portfolio of programs has grown will result in new challenges related to accountability that may not have been present (or needed) during the start- up phase. As a result, a more streamlined and transparent process for estimating and modeling program cost-effectiveness is needed. The study team traced Alliance claims of energy savings as presented in the MAR and developed alternative scenarios, which were subsequently run through the CE models. These analyses resulted in lower estimates of overall Alliance impact claims as compared to the numbers reported in the 2002 MAR (from 34aMW to 98aMW). Most all of this adjustment came in one program - the ENERGY ST AR(ii) Residential Lighting program - with adjustments in other programs examined being much less significant. The numbers used in this adjustment are from the "low influence scenario" for CFLs (see Chapter 3). THEME 5 : Benefits of the Alliance have exceeded costs. Even with the study team adjustment to estimated Alliance energy savings estimates, the analysis of program impacts shows the benefits from Alliance activities have exceeded its costs. In fact, the team analysis indicates a levelized cost for the Alliance portfolio of programs of between 0.83 and 1.17 cents per kWh, with a mean of 0.99 cents/kWh, when viewed from the Alliance perspective through the Venture period (see chapter 8 for additional detail on levelized costS). THEME 6 The regional approach of the Alliance is an asset and even greater leverage in program implementation can be gained in the future. The Alliance has developed an infrastructure of programs, relationships, and personnel that represents organizational capital that will be valuable in the future. Interviews with market actors indicated some diverse opinions regarding past efforts of the Alliance and the amount of energy savings that should be attributed to Alliance activities. Even taking those comments into account, there was a general consensus that the Alliance was able to undertake certain programmatic activities more efficiently on a regional basis than was possible through local efforts. The study team s review of programs selected for implementation by the Alliance indicated that most programs were well suited to implementation by a regional organization. In this respect, the Alliance was living up to its goal of focusing on market transformation projects that can best be addressed at a regional level. 2 It should be noted that the study team analyzed four programs from a retrospective perspective. Other than making an adjustment to the future consumer replacement cost of CFLs, no estimates of future costs, or estimated future savings were modified ftom Alliance estimates. Neither did the study team analyze local utility costs, or consumer O&M costs related to the programs. The cost estimates used by the Alliance for regional costs other than their own could have an effect on the levelized cost from a TRC perspective. Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report 2 Estimated Impacts of the Alliance Energy Savi ngs and Levelized Costs A key question associated with the analysis is how the results of the investigation into alternative scenarios for the four programs affect the overall Alliance investment perspective. To address this issue, the levelized cost from the Alliance perspective through 2002 was used. The calculation used for this is: Total $ spent to date on Alliance activities Levelized Alliance Savings to Date Using the Alliance numbers from the 2002 MAR for the costs ($96M) and savings (134 aMW) for all Alliance activities produces a levelized cost of 0.7 cents/kWh. The study team s analysis of the savings associated with the Energy Star Lighting, the Energy Star Windows, Building Operator Certification (BOC), and MagnaDrive programs when combined with Alliance estimates of savings for other tracked programs - produces an average cumulative savings through 2002 of 98 aMW compared to the Alliance s original estimate of 134 aMW. This analysis is documented in Chapters 3 through 6 of the main report and uses the low CFL attribution scenario from Chapter 3. These adjusted numbers (when summed with Alliance estimates for other programs) produce a study team estimated totallevelized cost for all the Alliance s activities of 0.99 cents/kWh, with a 90% probability that the cost is 1.17 cents/kWh or less, and a 10% probability that the cost is less than 0.83 cents/kWh. While this is an increase in the estimated levelized cost, it is still well below the avoided cost of power in the region. 3 Overall Value of the Alliance The interviews conducted with stakeholders and market actors contained a set of questions that addressed the overall value of the Alliance. Questions addressed four areas of Alliance activities - Planning, Implementation, Evaluation, and Communication. The individuals interviewed are shown in Appendix A to this report. They include project implementers, project evaluators utility program managers, retailers/trade allies, and other stakeholders. These questions were not asked of Alliance staff. The interviews with the evaluation personnel that conducted the MPERs was infonnative in that they were able to provide opinions based on their overall review of the programs and the interviews they had conducted with various market actors. The responses to these questions regarding the perceived value of the Alliance were combined with other infonnation obtained from the review of reports, and interviews with Alliance staff. This process produced insights that the study team believes are useful for this assessment. Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report E-4 Overall Value: Impact of Alliance on Market Transformation The interviews with stakeholders (excluding Alliance staff) indicated: A widely held belief that the Alliance is responsible for higher levels of market transformation than would have occurred had the Alliance not existed. That the Alliance had a significant impact on a number of markets for energy efficiency technology and their efforts on the energy efficient residential windows market in the Northwest is exemplary of a successful MT program. That the focus of the Alliance s market transformation efforts has been appropriate, i. the Alliance has done a good job of identifying and pursuing programs best addressed regionally. Also, the study team s reviews of the selection process and the programs implemented supported this finding. Examples include training programs such as builder operator certification (BOC), and projects targeted at manufacturers (e., Energy Star windows) as the types of programs whose implementation spans utility service territories and even state boundaries. While the comments above represent the study team s findings based upon the interviews, there were other comments made that did not appear to be part of the majority view, but the study team believed that they were worth bringing to the attention of the Alliance and its Board for their consideration. Two such comments were: There was one dissenting opinion regarding the impacts of the Alliance on market transformation which was based upon the belief that baselines are dramatically understated, i., many market changes observed would have happened without the Alliance - due to other factors. Another individual expressed the opinion that while venture selection has been good overall, there was concern expressed about the recent selection process. Overall Value: Alliance Tracking of Iml2!!:E1l There was a general view among t4e stakeholders interviewed that impact estimates tended to be modestly high. Comments pertaining to this finding include: Several individuals expressed concerns about the Alliance claimed impacts for its efforts in residential lighting and this did show up in the bi- modal estimates of CFL sales influence used in the study team analysis in Chapter 3. The individuals interviewed indicated that while they generally believe the Alliance impact estimates were somewhat high, it was believed to be only a modest overstatement on the order of 10 percent or so on average. There was a perceived need to update baselines used to estimate program impacts more frequently. There is a need to test performance assumptions in the field rather than assume that actual performance matches predicted performance. Evaluators did not feel pressured by Alliance study sponsors to produce favorable numbers or results. Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report In addition to the sets of comments that are viewed as study team findings above, other comments that may be of interest to the Alliance and its Board are: Some of the Board members interviewed (and also other stakeholders) indicated that they were "trusting" a few knowledgeable "other" Board members to ensure unbiased impact evaluations since they had expertise in this area. Originally the MPERs were designed to look at entire markets, not just at project indicators and effects; but, over time, they have changed to focus on project effects and delivery processes arrlless on the overall market. Overall Value: OrJ!anizational Effectiveness.A question was asked regarding how well the Alliance operated as a business organization. This was asked in the context of overall operational effectiveness compared to other organizations with which the individual being interviewed was familiar. Findings on this topic included: Only one interviewee rated the Alliance as "below average" as a business organization other respondents rated the alliance as well above average (7 or 8 on a scale of 1 to 10) and, based on other organizational studies, this is a high rating and reflects favorably on how the Alliance operates. Planning was viewed as a strength of the Alliance. Communication to stakeholders was rated as very good. Importantly, Alliance staff was well respected by the majority of stakeholders interviewed. Some other comments for consideration that were not viewed as findings are: Some concerns about recent trends in various areas (venture selection, accountability, and transparent accounting), but this was viewed as recent and not yet of great significance. The Alliance has faced recent challenges due to organizational changes in general, staff turnover and staffing patterns in particular. The Alliance s reputation among trade allies was raised as a "potential" concern - related to this concern was implementation contractor selection and their ability to relate to the concerns of trade allies and industrial customers in particular, and the implementation of an overall quality control process managed by the Alliance rather than its contractors to ensure that relations with key trade allies remained favorable. 4 Alliance Analysis Process and Issues Discussion A central aspect of this assignment was to consider alternative hypotheses concerning the market effects of Alliance programs. In meetings with the Ad Hoc Retrospective Committee, this was tenned the "But For" analysis and was meant to focus on detennining the appropriate baseline against which Alliance activities should be measured. In other words, a best practices Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report determination of what would have happened in these markets if the Alliance programs had not been offered. To address this question, a specific and targeted approach was developed to address alternative hypotheses and develop range estimates for both the baseline (i., what would have happened in the absence of the Alliance) and for "pivot" assumptions which focused on the assumptions or attributes of each program that had the greatest influence on estimates of program impacts. For lighting, pivot assumptions include the assumed operating hours per lamp, the savings per lamp installed, and installation rates. Each of these pivot assumptions was examined by the study team using a range of values that encoITpasses both a low and high scenario. The scenarios around these pivot assumptions were rolled up using a sampling framework to produce a range of likely impacts for each program and an associated likelihood of OCCUITence. The framework and approach to this "But For" analysis is documented in Chapter 2 and implemented in the detailed program analyses presented in Chapters 3 through 6. This process was used to produce the study team s estimates of Alliance impacts. Three Alliance processes were central to the study team s efforts to dimension the impact of the Alliance on the markets for energy-efficiency technologies and services. This included the cost- effectiveness analysis process, the evaluation process as represented by the MPERs, and the annual Market Activity Report (MAR) which presents the Alliance s estimates of annual and cumulative impacts. Select issues are discussed with each of these processes, then a set of recommendations is provided. The Alliance Cost-Effectiveness Model The study team conducted a review of the Alliance Cost-Effectiveness (ACE) model as part of the assessment of the Alliance s claims, particularly the models used for the four programs the study team investigated in detail. Difficulties encountered by the study team point to a need to streamline the ACE modeling process. Market Progress Evaluation Reports (MPERs)In developing the review of the Alliance accomplishments, the team reviewed a large number of the available Market Progress Evaluation Reports (MPERs). Since the objective of this assessment was to determine energy savings attributable to the Alliance and whether the Alliance has contributed to market transformation the study team s review concentrated upon how the MPERs can be used to determine the program progress. As a general statement, most MPERs focused primarily on assisting the Alliance on project implementation feedback, with some verification of market effects and a limited review of the input assumptions to the ACE model. Expected savings per application associated with a project and the related assumptions used in the cost-effectiveness model were not researched or not researched at the same level of detail. Overall, the MPERs are a good process review of program delivery, but did not research project impacts as actively as might have been expected. The Market Activities Report (MAR)The need for accuracy in the CE analyses and the role of the MPERs depends in part on what the Alliance and its Board hopes to accomplish with the energy savings estimates that are attributed to the Alliance in its MAR. The MAR is widely viewed as the Alliance s definitive statement on what it believes it has accomplished. The language used in the MAR supports this assumption. For example in the recent 2002 MAR, it Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report states that For the reporting period January 2002 to December 30, 2002, the Alliance has saved 45aMW, as depicted in Chart In the next paragraph of the MAR Executive Summary it states that Total Savings of the Alliance since its inception in 1997 (exclusive of utility direct rebates) 134aMW. " The Alliaoce energy savings claims are very direct and unequivocal. Given this, a question that must be addressed concerns the appropriate underpinning and evidence needed to ensure that these claims are viewed as credible by entities that work with and support the Alliance. The study team believes that it is important that the Alliance provide estimates of energy savings accomplishments, but there may be better approaches to bracketing and bounding the estimates than are currently used in the MAR. Even with that change, the MPERs should refocus a bit more on providing support for energy savings estimates. 5 Recommendations A number of specific recommendations are made by the study team in this section. The recommendations are divided into five categories and are presented below. Recommendation Area #1 Cost- Effectiveness Models and Processes: R1.I - The Cost-Effectiveness spread-sheet models should be "cleaned-up" and a better documentation process implemented to avoid confusion in the future. The study team believes that several rmn- weeks devoted to this process could considerably enhance the transparency, and user- friendliness of these important tools and save labor hours down the road that might well make up for the short term costs of upgrading the CE process. 3 RI.2 - Develop more specific processes to update and track assumptions used with direct links to sources of assumptions and referencing the MPERs that are tasked with reviewing the input assumptions for each proj ect. RI.3 - Board recognition of the complexity of this work element can help ensure that there is a reasonable review process for model results. The fact that a model is used does not in itself, ensure that the outputs are appropriate. Recommendation Area #2 Use of scenario analysis and identification of pivot factors in reporting of Alliance accomplishments: R2.1 -Evaluation and planning (e., the venture business plan) would benefit from the use of bounding scenario analyses and the identification of pivot factors. The portfolio committee discusses these factors when deciding whether a specific project should be undertaken, but these assumptions need to be documented and tracked over time. This would provide the following benefits: 3 There is the additional concern of knowledge being concentrated among a few key staff, if for some reason those staff members were not available to the Alliance in the future. Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report Large uncertainties on select pivot factors could be targeted in the MPER analyses to reduce these uncertainties. As the MPERs address these factors, better information can be incorporated into the CE analyses and they will better reflect the best available information as well as documenting current uncertainties. It would help determine the confidence in projected aMW accomplishments. Recommendation Area #3 - Treatment of Project Baselines (Dynamic versus Static): R3.1 - The baseline trend line for each Alliance project is determined in the planning process and not typically updated even when there have been substantial cha nges in the market. Outside factors such as changes in utility programs, prices, and energy shortages influence what would have happened without the Alliance program. As a result, baselines should be dynamic in that they should be re-evaluated every year and updated to reflect major market changes. R3.2 - The baseline is one of the most influential and uncertain factors in producing any estimate of Alliance project effects. Best efforts are needed on baseline determination, despite the complexities and uncertainties. Recommendation Area #4 - Trade Ally Relationships Going Forward: R4.1 -Trade ally relationships are central to the Alliance objectives. Implementation contractors represent the Alliance to these important stakeholders and some contractors have proven to be a key factor in program success. Additional independent quality control processes should be implemented to ensure that good relations are consistently maintained. Recommendation Area #5 - The Cost-Effectiveness Committee of the Board should revisit the way in which impacts are claimed and reported by the Alliance in public documents: R5.1 - Guidance is needed on what "claimed aMW impacts" and "levelized cost" means in the context of the MAR: Should the MAR only focus on market share indicators? , are these estimates meant to represent a "best estimate" of aMW attributable to Alliance activities? Should estimates be given a degree of confidence and/or expressed as range estimates to reflect the uncertainties in the attribution process? Should the MAR estimates be supported by a specific ACE analysis to ensure consistency between savings estimates and levelized costs? These positions have implications on Alliance credibility and also for resources allocated to evaluation efforts. In addition, this recognizes that all business decisio ns and venture analysis both within and outside of the area of energy efficiency investments are made with uncertainty, Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report and that precision in excess of what is used in the private sector to make good business decisions is not needed for the Alliance to successfully meet its objectives. 6 Final Comments This section is designed to emphasize the important context in which these issues are discussed and the recommendations made. The assessment perfonned by the study team found that the Alliance provides value that exceeds its costs. Interviews with key stakeholders indicated that the Alliance: Operates well as a business organization Is strong in planning, Communicates well, and Has impacted targeted markets. The benefit-cost analyses conducted with the study team s revised numbers show that the benefits of the Alliance has exceeded its costs. Overall, it is the study team s opinion that the reasons for establishing the Alliance are still valid and provide strong rationale for continuation: Energy markets invariably cut across utility and jurisdictional boundaries, it makes most sense to pursue these (MT) efforts regionally; and This regional approach by Alliance is an asset and can gain increased leverage by continuing its relationship building efforts with partners. Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report INTRODUCTION """, This report presents the results of the independent evaluation of market transfonnation accomplishments through the Northwest Energy Efficiency Alliance (Alliance) efforts since 1997. This retrospective evaluation effort was initiated by an ad hoc committee, consisting of Alliance board members and stakeholders representing non- board interests in the region, for the primary purpose of detennining whether the Alliance has transfonned enough markets over the first six years to justify the costs of the Alliance. This key question was investigated by looking at Alliance activities, and the role the Alliance has played in markets where the Alliance has spent significant resources over the past several years. This assessment of Alliance activities must also take into account the work of other organizations in these markets and their impact on energy efficiency to avoid double counting market impacts and effects. This evaluation was initiated in April 2003, and the work effort spanned approximately four months. 1 Background on Alliance History and Goals 2 Objectives of the Assignment 3 Project Activities 1.4 Layout of Report The evaluation team worked directly with the ad hoc committee chainnan and the executive director of the Alliance, and met with the committee at key project junctures to ensure that the evaluation was proceeding in a manner consistent with committee objectives. The team used the extensive body of material already available on Alliance project activities, including market assessments that were conducted on Alliance projects, market progress evaluation reports computer model results, and planning documents. Primary data collection activities focused on interviews with staff, evaluators, implementation contractors, market actors, and others to better understand program mechanisms and elicit alternative hypotheses regarding market changes. The results of the project are timely, considering that the funding cycle for the Alliance stakeholders requires renewal before the end of 2004. The evaluation team also provided observations from the analysis that can assist the 1\lliance in making improvements in. operations. In this chapter, Section 1.1 contains a brief overview of Alliance history and goals; Section 1. presents the objectives of this retrospective evaluation; Section 1.3 provides an overview of the activities conducted by the evaluation team; and Section 1.4 describes the organization of the report. Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report 1 Background on Alliance History and Goals The Northwest region has historically recognized the effects of conservation on the overall cost effectiveness of region s energy system. Substantial efficiency gains were realized during the 1980s and 1990s through utility conservation programs and the adoption of energy codes by state and local authorities in the region. In its 1996 draft power plan, the Northwest Power Planning Council (Council) assessed the remaining achievable potential for conservation and renewable energy in the four state NW regions. At that time, the Council estimated the amount of conservation that would be cost-effective to develop in the region over the next 20 years across a wide range of future electricity use patterns, gas prices, and hydropower availability. This analysis indicated the amount of conservation that would cost less than alternative power sources to be between 800 and 2300 average megawatts (aMW), with an average amount of approximately 1535 aMW.4 This assessment estimated the levelized costs for acquisition of this conservation resource to be approximately 1.7 cents per kWh. In the final report submitted that year, recommendations were made to fund a range of specific activities, including conservation, renewables, and low- income energy services.5 The report assessed a range of cost-effective conservation options, and made recommendations that all electric utilities operating within the region contribute to the development of conservation resources. The Comprehensive Review recognized that many energy efficiency efforts are most appropriately conducted at the local level, but that some conservation and renewable activities would benefit from regional planning and coordination. In addition to establishing a Regional Technical Forum to develop standardized protocols for verifying and evaluating conservation savings, the committee also recommended a coordinated effort to transform regional markets for efficient technologies and practices. The Comprehensive Review acknowledged that markets invariably cut across utility and jurisdictional boundaries, and thus it makes most sense to pursue market transformation activities regionally. They recommended that a nonprofit organization manage market transformation ventures for the region, and that the governing body should consist of consumer, utility, government, and public interest representatives. The organization should have a planned life of at least 10 years, in recognition of the time required to permanently transform markets and the range of markets or end uses to be targeted. A minimum investment standard for distribution utilities in the region was established, with some flexibility allowed in tre collection of these fees. The Alliance, which had recently been formed, was designated as the organization to manage the region s market transformation efforts. The Alliance had been formed specifically to see if more energy efficiency could be secured at a lower cost if the region pooled its resources to influence what was stocked and sold in the marketplace. In 1998, the Alliance hired Pricewaterhouse Coopers to conduct an operational audit to determine how well the Alliance was operating to achieve its objectives. This review focused on 4 Chapter 6 - Draft Fourth Northwest Power Plan, 1996.5 Comprehensive Review ofthe Northwest Energy System - Final Report "Toward a Competitive Electric Power Industry for the 21st Century," December 12, 1996. Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report organizational and administrative aspects of the organization. The assessment indicated that the Alliance s actions and plans were in alignment with the organization s purpose and objectives. The Alliance began printing annual activity reports and progress reports in 1999, the first set covering activities from inception through 1999, and subsequent activity reports produced annually. A refinement of the organizational goals was articulated in 2000, outlining the following goals for the Alliance 7 : Goal 1: The Alliance will rmnage a strategically prioritized portfolio of cost-effective market transformation (MT) projects. Goal 2: The Alliance will assist its projects achieve success by supporting related activities inc luding market research, information and education efforts, and partnering with other market actors. Goal 3: The Alliance will ensure that its ventures are evaluated to document the effects of its efforts and to use that information to improve future efforts. Goal 4: The Alliance will be an effective and open organization, and will keep its constituents well informed of Alliance activities. 2 Objectives of the Assignment A cost review plan that was adopted by Bonneville Power Administration in 1998 recommended that a review of the Alliance be conducted no later than 2004. In addition, contracts between the Alliance and its utility funders called for an independent review of the organization s market transformation accomplishments before the end of 2003 , in anticipation of funding reviews that will take place in 2004. In early 2003, the Alliance requested proposals to conduct this assessment, and outlined objectives for the assignment. 8 During the proposal process, and through a series of conversations with the ad-hoc committee members at the outset of this project, the following key objectives were identified: To determine if the Alliance has transformed enough markets over the first six years of its life to justify its costs. To provide an objective evaluation of accomplishments that can be given to stakeholders, and assist them in determining whether to continue investing in the Alliance. To determine whether Alliance claims for (aMW) savings are backed up, and to bound estimates of savings by dimensioning uncertainty around these estimates. 6 Northwest Energy Efficiency Alliance Operational Audit, Price Waterhouse Coopers, December 12 1998.7 Strategic Plan, Northwest Energy Efficiency Alliance, April 12, 2000.8 Request for Proposals for an Independent Evaluation of Market Transformation Accomplishments through Alliance Efforts Since 1997, January 2003. Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report To review project evaluations to determine whether they assessed the right things, and to suggest improvements to the evaluation process. To try and distinguish market transformation impacts from electricity price increases and technology advances. To identify where (which markets and activities) MT is the right tool, and areas where other approaches may be appropriate. 3 Project Activities The evaluation team organized the assessment into the following major task areas: Task 1:Kick-off Meeting, Scope Validation, Discuss and Finalize Work Plan Task 2:Review of Project Strategy and Market Indicators Task 3: Data and Information Collection Task 4:Scenario Development for Assessment (organization of data and information) Task 5:Analysis of Scenarios (Note: A scenario represents a set of Alliance accomplishments and resulting value. Task 6:Investment Assessment Framework (best estimates and ranges of Alliance results and value from a risk/reward perspective). Within each of these project tasks, a number of activities took place. A summary of the activities the team completed in conducting this evaluation is included in the table below: Activity Task 1: Kick-off Meeting, Scope Validation, Discuss and Finalize Work Plan Conduct project kick-off meeting and finalize work plan Develop project management and communications plan Submit initial data request Interview ad- hoc committee members and select board members Task 2: Review of Project Strategy and Market Indicators Review Market Progress Evaluation Reports (MPER) and Research Reports (approximately 100 reports) Conduct literature review on market transformation theory and practice Review Staff Recommendation Memos and Cost- Effectiveness analyses Review annual Market Activity Reports for 2001 and 2002 Develop interview guides and conduct initial interviews with key Alliance staff Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report 1-4 Activity Task 3: Data and Information Collection Initial review of levelized cost calculations Review of Progress Indicators for each ro' ect Submit additional data re uest Review ro ect s ecific documentation (im lementation re OrtS, curriculum, etc. Conduct ro ect u date meetin with Retros ective ad-hoc committee Review Alliance cost effective (ACE) models for programs selected for in-depth analysis Review ro rammatic actions and savin s occurrin in other re ions Task 4: Scenario Development for Assessment (organization of data and information) Determine the ivot assum tions for 4 in-de th rograms throu h sensitivity analysis Develop potential alternative hypotheses to test pivot assumption ranges and create discussion guides for market actor/stakeholder interviews Develop draft final re ort outline Conduct interviews with key market actors and stakeholders Consult other studies to assist in establishin ran es for ivot as sum tions Develo scenarios to establish distributions for ivot factors Task 5: Analysis of Scenarios Run ACE models for ke ro rams usin distributions for each ivot assum tion Anal ze interview data for common themes and otential recommendations Task 6: Investment Assessment Framework Best estimates and ran es of Alliance results and value from a risk/reward ers ective. 4 Layout of the Report Chapter two of this report provides a description of the evaluation approach used for this project. This is followed by four chapters that describe the results of the analyses for each of the four programs selected for in-depth analysis (Energy Star Residential Lighting, Energy Star Residential Windows, Building Operator Certification, and MagnaDrive). Chapter seven provides a summary assessment of other Alliance programs. A summary of the programmatic findings and an assessment of value to the region are contained in Chapter eight. Chapter nine outlines the conclusions and recommendations of the evaluation team, along with the key themes that emerged during the retrospective evaluation. The appendices of the report contain a) a list of those interviewed during the evaluation, and interview guides, b) distributions for pivot assumptions that were used in the ACE model runs and c) a list of Alliance documents used during the evaluation. Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report FRAMEWORK AND ApPROACH 1 Assessing Market Transformation Projects 2 Initial interviews with Alliance Staff 3 Approach to Program Analyses 2.4 Alternative Hypotheses Elicited and Explored 5 Approach to Scenario Analysis - Illustrated Example 6 Definitions Used in Program Analysis This chapter outlines the assessment issues addressed in this work effort, the framework used to define the assessment, and the approach taken to produce estimates of Alliance induced market impacts. Section 2.1 outlines the basic assessment problem for energy efficiency projects that use market transformation projects as the approach for encouraging cost-effective investments in energy efficiency technologies and practices by market actors. Section 2.2 discusses the initial interviews that were conducted to better define the assessment objectives. Section 2.3 presents an overview of the approach taken in this assignment. Section 2.4 discusses the processes that were used to develop alternative hypotheses concerning the role of the Alliance in promoting increased energy efficiency in the targeted markets, and Section 2. presents an example of the type of scenario analysis that was used as the basis for the in-depth project assessments. 1 Assessing Market Transformation (MT) Projects The Alliance focuses on MT as the "strategic tool... to encourage manufacturers, distributors and service providers to make affordable energy-efficient products available in the market place (2001 Annual Report). MT projects are typically developed in conjunction with a supporting theory and project logic that allow for the development of market indicators that can provide evidence" of a change in the market. Real market changes induced by MT programs rely on a variety of promotion, education, and incentive activities. Successful MT initiatives are by nature multi- faceted efforts that involve multiple organizations and evolving delivery mechanisms and progress metrics. At the Alliance MT ventures are designed with a focus on one or more of the following mechanisms; upstream training, entrepreneurial, or consumer. These mechanisms are discussed in more detail in Chapter 7 of this report. 9 Recent work done at the New York State Energy Research Development Authority (NYSERDA) utilizes this approach. Recent publications addressing these topics include "Chapter 3: Program Design Logic" in New York Energy $mart Program Evaluation and Status Report, September 2000; and also in "Chapter 5: Causality Approach" in New York Energy $mart Program Evaluation Report, January 2002. Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report In the context of theory-based or hypothesis-driven MT projects, a key component in evaluation is the verification of the validity of assumptions and hypotheses. The assumptions can be as straightforward as the assumed incremental savings associated with installing a high efficiency lamp, compared to standard efficiency options, and they can be complex when they address such things as the behavior of market actors (e., commercial architect, builder, buyer interactions). 1\ focused project theory and logic model construct can help identify pivot infonnation or assumptions, i.e., what must be true for the project to achieve its target benefits. These pivot assumptions often focus on three factors: 1) baseline assumptions, 2) the perfonnance of technology as it is applied in the field, 10 and 3) the inter-related issue of attribution/causality. Baseline issues involve both market conditions at the start of the intervention, and how that market will change over time without the intervention. Technology perfonnance will depend on field installation and operating characteristics. Attribution and causality issues can often be viewed as aspects of selecting the correct baseline, i.e., the ways in which the market has changed even if the project had not been offered and which changes can truly be attributed to the activities of the Alliance. It is nearly impossible to prove causality; instead approaches are focused on providing evidence of attribution/causality. From one perspective, this assessment can be viewed as an analysis of the investment in the Alliance and the return on that investment. There will be some elements that can be more easily quantified than others, but a thorough assessment will need to address the range of benefits produced by the Alliance. The approach taken by the study team develops risk/reward criteria based on scenario analyses of project outcomes, a method that is similar to assessments of investments in research and development portfolios by conducted by private companies. Initial Interviews with Alliance Staff 1\fter the evaluation team reviewed Market Evaluation Reports, and Research reports, and other documentation regarding Alliance activities and funding, one of the key steps in gaining a stronger understanding of Alliance activities involved interviews with Alliance staff. These interviews were focused developing a more complete understanding of the assumptions that drive each program. The key themes discussed during the interviews include the following: 10 Often project planners aSSUlre that energy-efficient technologies will be selected for application in those situations in which they will produce the greatest savings. However, this may not be correct. For example, research in the Scandinavian countries initially assumed that horizontal axis washing machines would be installed in households that use such machines heavily and therefore produce higher savings. Instead, field work showed that these machines tended to go to older, wealthier households and early project assumptions made about the usage characteristics of the machines as situated in the field were not borne out by the field work.II Discussions of causality in the context of project evaluation include Susser, M.Causal Thinking in Health Sciences: Concepts and Strategies, Oxford University Press, 1973; Hubennan, A. and M. Miles , " Data Management and Analysis Methods " in Collecting and Interpreting Qualitative Materials Denzin et al. eds., Sage Pubs. Thousand Oaks, CA, 1998; and Oakley, A. , " Confronting Causation Metascience, 12 149-152, 1997. Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report Role of project planning assumptions in development of strategy and theory. Understanding baseline assumptions and changes in baselines over time. Assumptions regarding in- field performance of measures. Selection of market progress indicators and ro Ie of indicators in supporting a market transformation story for that program. Role of the Market Assessment studies in providing a feedback loop back to implementation. Adaptive Management e., how are programs modified to reflect evaluation results and how are key choices made regarding such things as retiring certain programs or morphing existing programs into new offers. Begin the thought process regarding the appropriate way to explore alternative hypotheses regarding attribution of program results to ensure an unbiased view as called for by the ad hoc retrospective committee. These interviews were held at the Alliance offices during early June, 2003. At that time, the evaluation team was considering about a dozen programs for detailed review, and the results of these interviews assisted in narrowing the list of programs that would be 'representative' of Alliance activities. A sampling of the questions that were asked staff members during this phase of the project is located in Appendix A. 3 Approach to Program Analyses As part of the retrospective assessment of the Alliance s activities, a detailed investigation of four representative programs (chosen in discussions with the ad hoc committee) was undertaken. These four programs are Energy Star Lighting, Energy Star Windows, Building Operator Certification, and MagnaDrive. The process used to assess the Alliances claims for these four programs involved: Review of the MPERs Review of the most recent Alliance Cost-Effectiveness (ACE) model Threshold analysis Determination of the pivot assumptions Dimensioning uncertainty of these assumptions Simulating the results under different values for the pivot assumptions. The study team s review of the MPERs focused primarily on trying to obtain inputs that could be used in the investment and cost-effectiveness calculations; specifically measure savings (i.e., unit savings), costs, and penetration (i., number sold or installed). To determine program cost effectiveness, the Alliance has developed a model (Allia~e Cost Effectiveness) that essentially computes unit impacts and unit costs which are then fed into the NW Power Planning Councils ProCost model to develop various levelized cost and cost-effectiveness indexes. This model served as the foundation of the assessment of the Alliance s claims. It supplied information on Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report the assumptions used to develop these claims, and also was used to measure how alternative assumptions affect these claims. After reviewing the ACE model for each program, the evaluation team developed a list of the inputs used to compute the program s cost effectiveness. A threshold analysis was then conducted, which involves detennining how much the program s cost effectiveness changed in response to a change in each input. A list of key variables was developed based on how influential each input was on the final result. This analysis, combined with the review of the models and interviews, led to the development of a series of pivot assumptions for each program. These pivot assumptions represent the key inputs that may have values different from that assumed by the Alliance. Once the evaluation team identified these pivot assumptions, it then quantified the uncertainty about these assumptions by reviewing the literature and through stakeho Ider interviews. The approach for eliciting uncertainty estimates in the interviews is discussed in Section 2.4. Once the evaluation team had the probability distribution associated with each pivot assumption the next step in the assessment was to use tre appropriate ACE model to detennine how the programs accomplishments (savings, cost-effectiveness index, and levelized costs) where affected by the uncertainty in these assumptions. For this process, the evaluation used a simulation tool, ~Risk, and ran 5000 simulations of the ACE model which pulled observations randomly from the distributions developed for the pivot assumptions and observed the resulting effects upon the program perfonnance measures as defined by the Alliance. 4 Alternative Hypotheses Elicited and Explored Discussions were held with the ad hoc committee on the selection of a set of representative programs that provide an illustrative view of the Alliance s accomplishments. After detennination of the pivot assumptions, the next step in tre analysis focused on the elicitation of alternative hypotheses regarding changes in market activity, specifically focused on the four programs selected for detailed review. These alternative views were sought from a range of stakeholders and market actors. The objectives of these interviews included: tracing the quantitative assumptions used for pivot variables in the ACE models; exploring local, regional, and national effects other than the Alliance on the specific market being analyzed; and general feedback regarding the value and effectiveness of Alliance activities. In addition to implementation contractors, staff, and evaluators, the team sought input from retailers, utility reps, industrial energy managers, manufacturers, and national organizations that develop energy efficiency standards. The evaluation team provided the ad hoc committee chair and Alliance executive director with a list of individuals being considered for interviews and asked for additional suggestions. Both Alliance staff and tre committee provided additional suggestions for individuals who might share alternative views. In addition, the committee chair provided a 'letter of introduction' to potential interviewees that helped facilitate scheduling of interviews. A list of all individuals interviewed during the project is located in Appendix A. Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report 2-4 Interviews were scheduled and conducted in-person with key individuals, based on their availability in early August. Additional interviews were conducted via telephone after the on- site interviews. All interviewees were assured confidentiality in that no particular comment would be attributed to them directly, but their name and affiliation would be provided in this report. While these interviews do not represent a statistical sample, they do represent a cross section of viewpoints on Alliance activities. While some groups may appear under-represented the study team strived to get balanced input within the time constraints of the project, and availability of interviewees. When the stud y team determined that the information gathered on a specific pivot assumption was sufficient, when combined with other information resources, to confidently bound the ranges for that assumption, further interviews on that particular assumption were not sought. Each interview explored pivot variable ranges (when the individual had knowledge of the specific program variables being considered), alternative hypotheses, and a set of general questions regarding the focus and effectiveness of Alliance activities. Interviewees were given free reign to expound on any of the questions, and many insightful comments were elicited in this manner. The interview guides for the pivot assumptions for each of the four programs, and the general questions are located in Appendix A. The lists below indicate a range of 'starter' alternative hypotheses that were explored for each of the four programs. A number of individuals made qualitative comments relating to these initial or starter hypotheses, and other hypotheses were developed and explored as appropriate in the interview. This information was captured in interview notes and used by the evaluation team to explore specific issues further. The key objective of exploring alternative hypotheses was to dimension any uncertainty around Alliance claims, and analyze the "but for" hypothesis, i. what impacts would have happened anyway and which impacts could reasonably be attributed to the Alliance. Potential Alternative Hypotheses Explored during Interviews Enen!v Star Residential Liehtine 1. Energy crisis of 200 1 drove sales (energy costs and media awareness = indicators) 2. BP A and local utility coupon program spillover3. Field performance is not as anticipated (installation, removal rates, retention, ... 4. CFL stocking practices were driven by other market factors (availability, infrastructure) 5. Relationship to EP NEnergy Star programs drove sales Enen!Y Star Residential Windows 1. Baseline assumptions were different (nationally and regionally)2. Builders changed installation preferences for other reasons than Alliance activities 3. Manufacturers changed processes for other reasons than Alliance activities4. Distribution of electrically heated home is different than assumptions Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report Building,.Qperator Certification 1. Building operating practices baseline assumptions are different 2. Changes in habits of graduates do not produce savings assumed (per sq. ft. 3. Energy crisis of2001 (and resulting media attention) drove operator awareness and changed operating practices 4. Square footage managed by attendees was different than assumed. MaenaDrive 1. Baseline assumptions were different than assumed 2. The technology would have penetrated market w/o Alliance assistance3. Utility acquisition programs may have achieved similar results 5 Approach to Scenario Analysis - Illustrated Example The scenario analysis used in this assessment for the four programs selected for an in-depth examination was designed around the ACE Model that develops estimates of annualized savings using information on unit savings, units purchased or placed in the field, measure life, capital costs, annual O&M and other factors relevant for an economic assessment. The ACE model is based on the ProCost Model develop by the Northwest Power Planning Council (NWPC) and the inputs to the ACE model follow those developed by the NWPC and as part of the Northwest' regional technical forum. However, for any specific Alliance project, the inputs are unique to that project. A six-step approach characterized each program scenario analysis: Step 1:Begin with the cost-effectiveness analyses for the four programs identified for detailed analysis. Select pivot assumptions that influence cost-effectiveness. Trace assumptions to MPERs or other reference documents. Conduct interviews with other organizations to bracket impacts, key assumptions and develop scenarios. Step 5: Seek ranges for key values. Step 6: Delineate breakeven scenarios and distributions of economic outcomes. Step 2: Step 3: Step 4: The traditional approach to scenario development and sensitivity analysis typically involves picking a "best" or "most likely" case scenario, then high and low cases are developed to bound the results and test the sensitivity of the results to these alternative assumptions. An example to illustrate the traditional approach and the distribution approach is presented below using the Alliances Residential Lighting Project as ,a case study with which to illustrate each approach. Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report The traditional scenario analysis would pick a best estimate value and then high and low scenarios. Using the assumptions of the Alliance for its residential CFL project, this would translate into: Estimated Value : Alliance receives credit for all CFL sales that were not utility coupon or giveaway sales minus assumed baseline of 100 000 CFL sales (this baseline comes from the ACE model for the CFLs project) Low Scenario: High Scenario: The low scenario might assume that the many utilities in the region that developed their own CFL programs actually were the more important driver and that 30% of the CFLs that the Alliance is taking credit for in the Estimated Value" case are actually spillover from the utility coupon and giveaway programs to other sales ofCFLS, i., the awareness was created by the utility pro grams and that CFLs would have been available in adequate supply such that the utility programs were a more significant driver of total CFL sales than is assumed in the estimated value base case. The high scenario assumes that spillover goes in the other direction and that due to Alliance efforts, utilities are able to sell 30% more CFLs than would otherwise have been the case since the Alliance helped set up coupon programs, the redemption center and encouraged retailers to stock CFLs. The end result is that without the Alliance efforts the utility achieved sales would have been 30% less Translating these three cases into actual sales figures gives the following: A numeric example is used to illustrate two different approaches for perfonning scenario analyses - 1) the standard high, medium and low scenario approach, and 2) a distribution based approach. The numbers used here are for illustration only and are not the numbers that comprise the actual analysis of the lighting program presented in Chapter F or this example, assume that: 000 000 = Total CFL sales (Total) 900 000 = CFLs were sold with utility coupons or part of giveaways (Utility) 100 000 = Baseline, i.e. CFL sales that would have occurred without either utility or alliance efforts. The low, best estimate, and high scenarios defined above incorporate different assumptions about the direction of spillover, i., it is either zero + 30% or -30%. Thus, the three scenarios to be examined are: (1) LOW attribution scenario = (Total) - (Utility) - (30% spillover i., impact on non-utility sales cause by utility efforts) - (baseline) = 2.8 million (2) MEDIUM -- Estimated value = (Total) - (Utility) - (baseline) = 4.0 million. (3) HIGH attribution scenario = (Total) - (Utility) + (30% spillover, i., alliance efforts make utility sales 30% higher than would otherwise have been the case) - (baseline) = 5 170 thousand or roughly 5.2 million. Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report This scenario analysis provides three numbers that are obtained through the interviews with experts and using general reasonableness criteria. In summary, the Low estimate is 2.8 million CFLs attributed to the Alliance, the medium case is 4.0 million CFLs and the high is 5.2 million CFLs. What else would one like to know about these scenarios? Additional information that would be useful might include: . How likely is each of these scenarios to occur? Are scenarios other than these three as likely or more likely to occur? What is meant by low, medium and high? Is the low scenario the lowest conceivable value? Is the high the highest conceivable value? Just knowing these three values - a low scenario, a medium scenario and a high scenario -- may not tell us much and might not capture the expert judgment and ancillary information available very well The distribution approach used in the scenario analyses to be presented in Chapters 3 through 6 for the four selected programs attempt to increase the amount of information that is brought to bear on the problem. Specifically, when the interviews are conducted with experts, they are also queried about their opinion regarding the likelihood of the different outcomes. While it may be difficult to answer this question precisely, it is possible, for example, to have an expert tell you that he/she believes that the high scenario is more likely to represent what actually happened (in their opinion) than the low scenario. Asking general questions about the odds of the high and low scenario occurring may produce information such as that shown in Exhibit 2-1 below where the medium scenario is believed to be the most likely, the high scenario is believed to be more likely to occur than the low scenario (by a ratio of 3:2) and there is also a small possibility that the true outcome is either above the high or below the low scenario. Exhibit 2-1 portrays a distribution the embodies this additional information Exhibit 2- Example of Distribution-Based Scenario Analysis 10% Distribution for Scenario 40% 30% 20% Prob. Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report The assessment of the likelihood of occurrence for the different scenarios adds additional, useful information. Individuals familiar with the market can provide "best" available information on the relative likelihood of occurrence. While only incorporating rough estimates of the likelihood of occurrence, the end result is a better representation of the scenarios being assessed. To complete this example, distributions are developed for select pivot factors that drive the final estimates of impacts and attribution to Alliance efforts. In the case of the lighting project, two distributions are developed from judgment, expert opinion and augmented by secondary research (e., other lighting evaluations). These two distributions are: I. Number of lamps sold due to Alliance activity. 2. Savings in Watts for each lamp sold (takes into account installation, retention, wattage and other factors). The final distribution for overall attributed impacts is developed by: I. Taking 5 000 "random draws" of values are made from each distribution; 2. For each value from each distribution, the final attribution value is calculated for that set of drawn values 3. This gives us 5 000 values which are graphed to give us the final distribution of impacts attributable to the alliance. The following Chapter 3 illustrates this process used for the Alliance Residential Lighting Project, and subsequent chapters present results for Alliance Windows Project, Builder Operator Certification Training, and for the MagnaDrive project. 6 Definitions Used in Program Analysis The following terms are used throughout the report to describe the quantitative effe cts of Alliance activities. Several of these are defined in the ACE model. Venture period: Savings are based on the actions that occurred from inception through 2003, and they include continued savings each year through 20 10 for those actions implemented during the venture period (i., the 'demand side power plant' is built and will continue to produce savings 12 Source: "Alliance Portfolio Cost Effectiveness (2002 Numbers)" documentation to "2002-Alliance-Project- Benefits-Summary-4-14-2003.xls" spreadsheet Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report Cumulative Savines throu2h 2002 The energy savings in average megawatts accomplished from program inception through the end of 2002 (but not including future savings streams) Alliance Perspective (C-E and B/C Ratio) Costs are only Alliance dollars spent Benefits are regional aMW achieved with the addition of 7.5% T &D losses. Total Resource Cost Perspective (C-E and B/C Ratio) Costs are regional costs ("Alliance, local utility administration costs, consumer first costs consumer O&M, etc. Benefits are regional aMW achieved with the addition of 7.5% T &D losses plus the total regional ancillary benefits ("T &D deferral credit, water savings, natural gas savings etc. " Levelized cost The present value of a resource s cost (including capital, financing, and operating costs and quantified non-energy benefits) converted into a stream of equal annual payments and divided by the annual kWh saved.,,13 13 Source: 2001 Market Activities Report (MAR) Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report ANALYSIS OF EN ERGY STAR RESIDENTIAL LIGHTING This chapter presents the in-depth evaluation of the residential lighting program. Section 3.1 provides a brief background on program goals, mechanisms, and activities. Section 3.2 describes the process used to evaluate and bound the impact estimates for the program, and compares the results with Alliance claims. Section3.3 presents the findings that emerged from the assessment of the lighting program. 1 Introduction to Program 2 Assessing Program Accomplishments 3 Assessment Findings 1 Introduction to Program The Alliance initiated a strategy to transform residential lighting markets in 1997 by creating two regional programs in that year, one focused on lamps and the other on fixtures. These two programs (Lightwise and Energy Star Residential Lighting fixtures) were combined in 2000 under the Energy Star (ES) platform. While early program efforts had worked with manufacturers to accelerate availability of high quality compact fluorescent lighting (CFL) products for the consumer marketplace, the program began shifting its eIIphasis to a marketing strategy in the retail arena. The program strategy is to promote efficient residential lighting through the Energy Star technical specifications and marketing messages. Overall program goals are: Encourage consumer purchases of new generation CFL technology Coordinate and leverage utility efforts to promote Energy Star products Encourage the development of new energy efficient lighting technologies Expand the use of Energy Star fixtures in new construction. The Northwest saw a dramatic increase in sales of CFLs during 2001 , partially driven by the energy crisis in California and the resulting media coverage, utility coupons, rate hikes and retail advertisements. 14 The Alliance CFL program played an active role in facilitating the jump in interest in CFLs through development of cooperative efforts between the Alliance, BP A, the region s utilities and retailers. The program adapted to the changing market conditions in the NW by coordinating marketing messages and materials for retailers, and through outreach to new partners in the region. The implementation contractor for the Alliance, Ecos Consulting, also 14 Saturation, Penetration, Transformation: How Do You Know When a Market Has Changed?, Stephen GroverDavid Cohan, and My K. Ton Teaming/or Efficiency, ACEEE Summer Study on Energy Efficiency in Buildings 2002. Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report managed the coupon program funded directly by BPA and some of the region s utilities during the same time period. While the coupon call1'aign was not part of the Alliance Lighting Program, it did take advantage of the retailer network already established by Ecos and the 1\lliance. The Alliance has worked through its program cooperative agreements to develop a tracking system that accumulates quarterly sales data for participating retailers in the region. These data are then used as the basis for estimating sales at non-participating retailers, and total sales in the region. When data regarding coupon sales and utility give-away programs are backed out of total regional sales, the Alliance estimates of CFL sales attributed to program efforts remains. Specific progress indicators for the program include: Increase consumer understanding and awareness of Energy Star products Increase availability and variety of Energy Star products at retail Increase sales of CFLs to first time buyers Increase consumer satisfaction with CFL purchases Increase promotional efforts by retailers to sell Energy Star products Price of Energy Star products continues to drop. Total Alliance savings claimed through 2002 for the program are 70.4 aMW. 2 Assessing Program Accomplishments This section assesses the accomplishments of the Alliance s Energy Star Lighting program in terms of measurable performance metrics. Based on the market activities report (MAR), the evaluation team defines these metrics to be the program s electricity savings (in aMW), its cost- effectiveness from the Alliance s perspective, as well as the total resource perspective, and the levelized cost (from both perspectives). As discussed in Section 2.3 , the assessment involves determining the key assumptions underlying the program s assumed accomplishments developing alternative hypotheses based on these assumptions, and running these scenarios within the Alliance Cost-Effectiveness model to determine their impact. The sections below present a detailed discussion of these steps. This is followed in Section 3.3 by the results of this assessment. Pivot Assumptions The first step in assessing the accomplishments of the ES Lighting program is determining the key assumptions required for quantifying the chosen metrics. While there are many assumptions involved in assessing a program, the evaluation team restricted its attention to those inputs that 15 Northwest Energy Efficiency Alliance Residential Energy Star Lighting Progra m, Annual Report 2001-2002, by Ecos Consulting. 16 This number represents the Alliance estimate of savings as reported in the 2002 Market Activities Report. Note that direct utility /BP A rebates were removed from the total lamp savings for the region (the amount backed out for rebates was 32 aMW in 2001, and 2.7 aMW in 2002). Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report are subject to a relatively high degree of uncertainty or had a significant impact on the outcome. These key assumptions are denoted as "pivot" assumptions. For lighting, the pivot assumptions can be broken down into two groups: 1. Assumptions required to comp ute the annual kWh savings associated with each CFL sold. 2. The number of CFLs sold due to the efforts of the Alliance. Based on previous experience with other lighting retrofit programs and a review of the ACE model, the evaluation team determined that the key pivot assumptions involved in computing the annual kWh savings associated with each CFL are: The displaced wattage (the difference in wattage between the new CFL and the incandescent bulb being replaced). The hours that the CFL is used each day (which differs between interior and exterior bulbs ). The average lifetime (in run-time hours) of the bulb. The installation and removal rate. This includes both those CFLs that were purchased but not installed and those bulbs that were removed and not replaced with another CFL. The price of the bulb (which does not impact the kWh savings, but is important for the cost-effectiveness of the program from the total resource perspective). There are essentially two pivot assumptions for the number of CFLs sold due to the efforts of the Alliance. The first pivot assumption involves determining how many of the over 8 million CFLs sold in 2001 was due to the efforts of the Alliance versus how many were due to the California energy crises, the BP A coupon program, and utilities' giveaway programs. The other pivot assumption is to what degree this explosion in CFLs sales affected the baseline (i., non- Alliance influenced sales) going into the future. Once detennined these pivot assumptions, the next step in the analysis was to develop and quantify alternative hypotheses for these assumptions. This task is discussed in the next section. Alternative Hypotheses After identifying the pivot assumptions associated with the ES Lighting program accomplishments, the next step is to identify both meaningful alternatives to these assumptions and their likely occurrence (i., their probability distribution). In this section the evaluation team presents the values for its alternative hypotheses and the source of this information. It is important to note that in general the pivot assumptions associated with the kWh savings per bulb are generally measurable, and there is a fair amount of research on these assumptions. However the pivot assumptions associated with the numbers of bulbs sold due to Alliance influence is based on the judgment of the experts interviewed and information contained in the evaluation reports designed to address the attribution of effects to the Alliance activities. It is possible in an evaluation activity to ask market participants whether actions of the Alliance influenced a market action and to what degree. Overall, the goal was to gather and use the best available information on the identified pivot assumptions. Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report Displaced Wattage The Alliance, in their ES Lighting Cost-Effectiveness model, assumes that the displaced wattage associated with each CFL is 74 watts. This was based on a survey conducted as part of the Lightsaver program. This survey showed that approximately 23% of the bulbs in this program were 30 watt CFLs replacing 150 watt incandescent, and an additional 33% where 27 watt CFLs replacing 100 watt incandescent. Based on information obtained from several sources, 17 the evaluation team determined that an alternative assumption would have, on average, a displacement of 58 watts. The evaluation team further assumed a distribution of possible displaced wattages where 80% of the observations would lie between 47 and 70 watts. Hours of use The Alliance assumed that the new CFL would be used for 3 hours in an interior location or 5 hours in an exterior placement. Based on several of the same resources cited for displaced wattage 18 the study team assumed that the hours of use would average 2.75 hours interior and 4 hours exterior. The team further assumed that while there remained some uncertainty about these numbers, 80% of the possibilities would occur between 2.1 and 3.3 hours per day for interior lamps and between 2.4 and 5.5 hours per day for exterior lights. These numbers reflect some amount oftakeback that often occurs when customers install more efficient lighting. Lifetime Based on the review of manufacturer s literature, the evaluation team found that advancements in production techniques have resulted in an increase in the expected life ofCFLs. A wide range of lamp quality and claims are made within the industry. As an alternative hypothesis, the evaluation team assumed that the average lifetime has increased from the Alliance s 7 000 hours to an average of 7 500 hours, with 80% of the probability going from 6 000 hours to 9 000 hours. 17 A number of studies were consulted during this analysis, including the following: S. Lighting Market Characterization - Volume 1: National Lighting Inventory and Energy Consumption Estimate - Final Report" Navigant Consulting for U.S. DOE EERE, September 2002; indicates installed lamps average 67 watts for incandescent, and 18 watts for CFLs. A recent utility program evaluation in the NW "Conservation Kit Program Evaluation" Seattle City Light - May 2003 presents information that appears to indicate an avg. displaced watts - 57 watts. An older utility report completed in the region , " Impact Evaluation ofMPCs Residential Lighting Program, Hagler Bailly, December 1995; used the following values to estimate displaced watts = 62 (1993 program) and 59 (1992 program). One major retailer indicated the majority of the bulbs sold were 60 watt replacements typically a 13 watt CFL resulting in a 47 watt displaced wattage estimate. While it is acknowledged that the estimates used by the Alliance were developed by the NW Power Planning Council, and generated at a time when the program had a significant mixture of fixtures and bulbs - with a focus on bulbs in high-use areas, since the majority of the claimed program savings occurred in 2001-, the evaluation team based its distribution of displaced wattage on these and other studies, interviews conducted, and professional judgment.18 The recent DOE study uses 2.1 h/day for exterior lighting, and 3 h/day as the highest interior level for one room. (2.2 used as CFL avg.). The MPC study from 1995 used 4 h/day (based on ELCAP 3.75 h/day data from 1992 - and primary survey data of 4 h/day). Based on these data, and the input from those interviewed, the team selected the mid-point numbers of2.75 and 4.0 hours for interior and exterior lighting, respectively. Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report 3-4 Installation/Removal Rate The original Alliance Cost-Effectiveness model assumed that 1) all purchased CFLs would be installed, and 2) 12% of the installed lamps would be removed by the customer within the first year and not replaced. This figure was based on survey responses reported in a recent MPER. A more recent presentation by the Alliance s evaluation contractor indicates that, based .on survey data, one year after purchase about 85% of the lamps sold in wave 1 were still installed, and 76% of those sold in wave two were still in place. 19 Additional research and interviews suggest that that not all purchased CFLs would be installed, particularly given that many of the bulbs sold in 2001 and 2002 were in multi-lamp packs.2O Therefore, the evaluation team assumed a combined average installation and removal rate of 28% (meaning 72% of purchased lamps were still installed one year after purchase), with the range of 16010 to 40% that brackets 80% of the observations for removal of lamps. Price per Bulb With the increase in CFLs from foreign manufacturers and the increased stocking of these bulbs in large discount retailers such as Costco, the evaluation team determined that the cost of a bulb assumed by the Alliance ($8.00 in 2003 and $6.00 in 2007) may be high, and a more realistic range of prices may be $4.40 to $5., with an average of$5 in 2003, and a range from $2.40 to $3., with an average in $3.00 in 2007. Savings per Lamp The net result of these alternative hypotheses on the kWh savings per bulb is a change from the Alliance s original 66 kWh per bulb down to an average of 39 kWh per bulb. 21 Exhibit 3- shows the distribution about this result based on the probability distributions of the underlying pivot assumptions. It is noted that, independently of this evaluation, the Alliance recently updated projections associated with savings per lamps based on input from NWPPC, and will be using approximately 39 kWh/lamp/year to project savings for the CFL program going forward. 2001 Sales due to the Alliance In 2001 , sales ofCFLs in the region increased significantly, from 650 000 lamps to over 350 000 lamps (based on Alliance data). This large jump was due, in part, to the California energy crisis, a BP A coupon program, and a free lamp giveaway program sponsored by several 19 "Residential Lighting Program Market Progress Evaluation Report " Stephen Grover, EcoNorthwest, presented to Alliance June 16, 2003. 20 "Conservation Kit Program Evaluation: Transforming the Residential Use of Compact Fluorescent Lighting. Tachibana, D.LO., Seattle City Light, May 2003. In this giveaway program, 92% of participants installed at leastone bulb from the kit, and 67% installed two, immediately after receiving the kit. Including later installation, and net of free riders, 1.7 bulbs per kit were installed. Among kit participants, 66% tried a CF bulb in their home for the first time. Impact Evaluation of MPCs Residential Lighting Program , RBI, December 1995 used a 79% installation rate after 1 year. 21 Interview notes indicate that PGE uses 55 kWh/year for their estimate of per lamp impacts. Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report area utilities. The 2001 MAR states "In 2001 , the region captured 31 aMW of savings through market effects including Alliance programs, utility marketing campaigns, price elasticity's and Exhibit 3. Distribution for Annual Unit Savings (kWh)/D86 060. 050 Mean=39.3301 040 030 020 010 000 .. 80 10% 28.0143 51.8536 10% retailer and manufacturer marketing efforts." When CFLs that were purchased with utility or BP A coupons and those given away by utilities are subtracted, about 4.2 million lamps remain. The pivot assumption in this area is the number of these bulbs due to the Alliance s ES Lighting Program. The Alliance cost-effectiveness calculations assume that all of the bulbs that are not purchased by a coupon, or given away by utilities are credited to Alliance activities. This results in over 4.2 million CFLs being credited to the program in 2001. Since the evaluation team cannot directly measure who was responsible for each bulb sale, it relied on responses in its interviews from retailers, utility program managers, and other knowledgeable individuals to develop an estimate of the 2001 sales due to the Alliance. The team consistently ran into two divergent opinions. One was that the Alliance was responsible for these 4.2 million bulbs and may even be responsible for some of the remaining 4.1 million bulbs because of their work in developing the necessary infrastructure to support such a large demand for CFLs. The other group believed that the Alliance was not as influential in the market, and was responsible for a significantly smaller fractiJn of the total sales.22 In addition to input from those in the region on this attribution question, the evaluation team can compare changes in sales volume in the NW to what occurred in other parts of the country during the same time period. Nationally, sales ofCFLs tripled between 1999 and 2001 , and then almost doubled again in 22 One major retailer even suggested that the Alliance had little to no effect on their sales of CFLs, and another interviewee felt that utilities that ran coupon programs in the region should be credited with spillover for bring people into stores. Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report 2002. While the NW may have seen sales number increase tenfold from 2000 to 200 I, sales increased four- fold in California during the same period. To accommodate the disparate regional views, the evaluation team constructed two alternative scenarios. The "high influence" alternative assumes that on average, the Alliance was indeed responsible for 4.2 million CFL bulbs in 2001 , and may even be responsible for some of the coupon sales. The other assumption is that the Alliance is only responsible for half of the non- coupon sales, or approximately 2 million bulbs. The assumed distribution for each alternative is presented in Appendix B. 2002 Baseline Given the huge increase in CFL sales in 2001 , the next pivot assumption is the degree to which the lighting market has been transformed by this event; i.e., What are the baseline CFL sales going into the future? Before the 2001 crisis, the Alliance assumed that the baseline sales of CFL were around 40 000 per year, and after this event the baseline increased to 100 000 per year. Following the two alternative viewpoints presented above, the evaluation team again assumed a high influence" baseline, which was consistent, on average, with the 100 000 number. The "low influence" baseline assumed that the baseline for 2002 going forward was increased to 200 000 units, on average. Exhibit 3-2 summarizes the above values the evaluation team assumed for the pivot assumptions as well as the value for these assumptions used by the Alliance in their cost-effectiveness model. Exhibit 3-2: CFL Pivot Assumptions Assumption Alliance Team Displaced Wattage 74 watts 58 watts Hours on Per Day 3 interior 75 interior 5 exterior 4 exterior Lifetime 000 interior 500 interior Installation/Removal 12%28% Price per Bulb $8 ~ 2003 $5 ~ 2003 $6 ~ 2007 $3 ~ 2007 Alliance Influence on 2001 Sales 253 827 High Case: 4 261 314 Low Case: 2 064 454 2002 Baseline Sales 100 000 High Case: 100 000 Low Case: 200 000 KWhlbulb/year 23 "Market TransfonTIation: Substantial Progress from a Decade of Work," Steven Nadel, Jennifer Thome, Harvey Sachs, Bill Prindle, and R. Neal Elliott American Councillor an Energy Efficient Economy, Report Number A036 April, 2003. Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report 3 Assessment Findings This section reviews the results of the simulation of the Alliance Cost-Effectiveness model using the above alternative hypotheses on the pivot assumptions. Following the structure of the MAR the evaluation team focuses on the impact of these alternative hypotheses on three main areas: The cumulative savings (aMW) associated with the program The levelized cost (cents/kWH) from the Alliance and total resource perspectives, and The cost-effectiveness ratio, also from the Alliance and total resource perspectives. To determine these impacts, the evaluation team used the ACE modef4 appropriate for this program, and altered the input assumptions as discussed above. Using a Monte Carlo simulation tool, ~Risk, the evaluation team ran 5000 simulations of the ACE model, which pulled observations randomly from the distributions developed for the pivot assumptions. Since this is a retrospective review of the Alliance s performance, the evaluation team focused on the implication of these alternative hypotheses up to through the venture period, as defined by the Alliance. Cumulative Savings The impact of these alternative scenarios on the cumulative aMW savings through 2002 due to the ES Lighting program is presented in Exhibit 3-3. for the high influence scenario and Exhibit 3-4 for the low impact scenario. Exhibit 3- Distribution for 2002 Cumulative aMW/F72 04Oi 00010 35 L10% 27.8553 55.867 110 10% 24 CE-C97-023C-ES-Lighting-Post-Crises-Ext.xls, with a run date of June 7, 2002. Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report F or the high influence scenario, the cumulative savings through 2002 drop from the Alliances estimate of 70.4 aMW to a mean value of 40.8 aMW. This drop is due primarily to the decrease in the annual kWh savings per bulb. The savings at risk2s at the 80% quantile is approximately 13 aMW. For the low influence scenario, the cumulative savings to 2002 drop further to 30.3 aMW, with a savings at risk of10 aMW. This implies a 10% likelihood that the cumulative savings were below 21 aMW for the CFL program. Exhibit 3-4 Distribution for 2002 Cumulative aMW /F72 0601 10% 20.9777 40.5898 100 10% Levelized Cost Exhibit 3-5 presents the resulting average levelized costs from the Alliance and Total Resource perspective for ES Lighting program under the high and low scenarios. While the alternative hypotheses in both cases has increased the levelized cost relative to the Alliance s original results, the levelized cost for both the high and low influence cases are well below the avoided electricity cost for the region. Cost-Effectiveness Exhibit 3-6 presents the resulting average cost-effectiveness from the Alliance and total resource perspective for ES Lighting program under the high and low scenarios. As expected, given the levelized cost results, the program is still cost effective, though in the low influence scenario the CE ratio is slightly below 1.0 from the Total Resource perspective during the venture period. 25 The savings at risk is the loV\est expected savings under these assumptions at a given level of probability. Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report Overall, regardless of the assumptions used (displaced watts, attribution of savings, etc), the program is still cost effective and has a levelized cost well below the cost of power in the region. Exhibit 3-5: Average Levelized Cost, ES Lighting LEVELIZED COST Alliance Team (cents/kWh)High Low Alliance Perspective Venture + Post Period 0.15 Venture Period Only TR Perspective Venture + Post Period 1.04 1.2 1.8 Venture Period Only 0.44 1.9 Exhibit 3-6: Average Cost-Effectiveness, ES Lighting Cost-Effectiveness Alliance Team Index High Low Alliance Perspective Venture + Post Period 27.12.10. Venture Period Only 17. TR Perspective Venture + Post Period 1.5 1.2 Venture Period Only 2.1 1.2 1.0 26 Source: 2002 Market Activities Report, Northwest Energy Efficiency Alliance for TR perspective venture + post venture period numbers, and cumulative savings numbers (70.4 aMW) reported as Alliance claims. All venture period numbers were pulled from the ACE model provided to the team. The ACE model was updated by the Alliance after 2001 to reflect actual CFL sales in the region, and re-run to produce adjusted levelized cost nunDers. For 2002 savings reported in the MAR, the ACE model was not updated by the Alliance, thus the savings numbers used in the ACE model are based on predicted 2002 sales, not actual, and the levelized cost numbers reported in the MAR do not reflect the updated aMW savings as reported in the MAR. As described in chapter 2, the evaluation team used the ACE model for all program comparisons. In the case of the lighting program, this results in teamestimates of program savings and levelized costs that are consistent with the ACE model, but not necessarily consistent with MAR-reported savings numbers. 27 Ibid. Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report Qualitative Assessment The evaluation team s review of Alliance documents and interviews conducted for the evaluation provided numerous insights on the CFL program, and also helped the team bracket the savings estimates. A summary of relevant comments and findings is provid ed below: The Alliance has contributed to increased awareness of CFL technology and the Energy Star brand in the Northwest over the past several years, and thus was a key contributing factor to the overall increase in sales in the region in 2001-02. Recert sales data suggest the market for CFLs in the region will not drop back to pre-crisis levels, but an adjustment to baseline levels should be considered by the Alliance to reflect post-crisis conditions. The Alliance s collection of actual sales data from a majority of the region s lamp retailers is commendable. Ongoing, innovative, co-marketing activities with major retailers are important to the future success of the program. The Alliance has a reputation for bringing new ideas to the table when working with participating retailers. It is important to assure that ongoing marketing activities meet the needs of a broad cross section of retailers. A number of comments were made regarding CFLs being used in the wrong applications by many customers, leading to dissatisfaction with lamp perfonnance. Both consumers and retailers must be better educated on proper selection and use of CFLs. At this stage, the Alliance does not have a clearly defined exit strategy for residential lighting. The Alliance should strive to define when the market has been adequately transfonned. This could be defined as reaching a market share of all lighting sales, or another related metric. 28 The Alliance did not conduct a coupon program in the region, but the program implementation contractor managed these programs for BP A and some regional utilities. The use of a coupon fulfillment process for administering lamp rebates is expensive. Most large retailers now have sophisticated inventory control systems built into their operations that allow for accurate tracking of lamps (or other products) sold. For future resource acquisition programs, these data could be used to simply reimburse retailers for Energy Star lamps sold. This process has been used by some utilities in California. Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report ANALYSIS OF EN ERGY STAR RESIDENTIAL WINDOWS This chapter presents the in-depth evaluation of the Energy Star Residential Windows program. Section 4. provides a brief background on program goals mechanisms, and activities. Section 4.2 describes the process used to evaluate and bound the impact estimates for the program, and compares the results with Alliance claims. Section 4.3 presents the findings that emerged from the assessment of the Windows program. 1 Introduction to Program 2 Assessing Program Accomplishments 3 Assessment Findings Introduction to Program The Alliance Windows program began in 1998 amidst a focus by the Alliance on building a positive image among market actors in the residential sector for the Energy Star brand in the Northwest. In the spring of 1998, the Alliance rolled out the Windows project and began meeting with builders and window manufacturers and attending trade shows in the region. The implementation contractor for the project had worked with the National Fenestration Rating Council on national standards for windows. This background allowed for the development of a strong working relationship with manufacturers in the region. Because a high percentage of windows sold in the region are also manufactured in the region, the project focused on working closely with six of the largest manufacturers in developing less costly methods of building high-efficiency windows. Using alternative technologies, the project was able to demonstrate that low-e coatings, stainless steel spacers, and better frame design would allow manufacturers to meet new U-value targets without significantly increasing costs. The Alliance Windows project initially set a target U value of 0.30. When the national Energy Star program set its standard of 0.35 , the Alliance recognized the value of maintaining consistency with the national marketplace, and adopted 0.35 as their target as well. This would allow the Alliance to leverage national marketing materials in their partnerships with Northwest organizations. Project partnerships were expanded to include co- marketing activities with utilities, retailers, and manufacturers. The Alliance was able to leverage Energy Star dollars through matching funds provided by manufacturers and other partners. Throughout the project the Alliance conducted retailer training, and worked with state and local governments to establish prescriptive requirements in building codes for window U values. 29 More detailed program history and a review of recent activities can be found in Energy Star Windows, No. Quantec, January 2002. Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report Market penetration for Energy Star Windows continued to climb rapidly, and project targets were exceeded by the end of2000. In June 2001 , the program ended. In summary, the program strategy was to build product image and brand association to increase the sales of high efficiency windows so that they would become the nonn in the Northwest market. Program delivery mechanisms are described briefly above, and can be summarized as: Conduct promotional initiatives through co-marketing arrangements to leverage national marketing messages. Work with manufacturers to develop more cost-effective methods of building energy efficient windows. Provide training materials and technical assistance to retailers, builders, and code officials in the region. Educate consumers on the benefits of high efficiency windows. Progress indicators for the program include: Cooperation from a majority of the window manufacturers in the region in tenns of significant cost sharing of marketing. Increased awareness of Energy Star fenestration products and their benefits. Increased market share (up to at least 54%) of Energy Star fenestration products in all four states. Total Alliance savings claimed through 2002 for the program are 13.8 aMW. Energy Star windows achieved a 70% market share in the Northwest by the end of 2002. 2 Assessing Program Accomplishments In this section, the evaluation team assesses the accomplishments of the Alliance s Energy Star Windows program in tenns of measurable perfonnance metrics. Based on the MAR, the evaluation team defines these metrics to be the program s electricity savings (in aMW), its cost- effectiveness from the Alliance s perspective as well as the total resource perspective, and the levelized cost (from both perspectives). As discussed in Section 2., the assessment involves detennining the key assumptions underlying the program s assumed accomplishments developing alternative hypotheses based on these assumptions, and running these scenarios within the Alliance Cost-Effectiveness model to detennine their impact. The sections below present a detailed discussion of these steps. This is followed in section 4.3 by the results of this assessment. 30 2002 Market Activities Report. Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report Pivot Assumptions Following the approach outlined for ES Lighting, the first step in assessing the accomplishments of the ES Windows program is determining the key assumptions required for quantifying the chosen metrics. While there are many assumptions involved in assessing a program, the evaluation team restricted its attention to those inputs that are subject to a relatively high degree of uncertainty or had a significant impact on the outcome. These key assumptions are denoted as pivot" assumptions. One of the pivot assumptions for this program is the savings associated with the installation of ES windows. Determining these savings is a complicated process, and is dependent on characteristics of the buildings where the windows are installed, the difference in efficiency between the Energy Star product and the product being replaced, and weather conditions. It was beyond the scope of this project to conduct an analysis of the energy savings model being used to estimate savings associated with the windows, so the evaluation team assumed that the values used by the Alliance in their ACE model are correct. 31 The number of ES windows installed each year is also another pivot assumption for this program. Estimates of market penetration were based on data provided by manufacturers to the program implementer, and on through triangulation of surveys of retailers, wholesalers, and builders conducted by the evaluation contractor. The evaluation team did not perform an uncertainty assessment on these data. The ACE model itself is driven off 1997 information on window sales. The remaining variables that affect the outco~ of the assessment are electric heat saturation and incremental cost. Therefore, for this analysis, the pivot assumption the evaluation team investigated are: The proportion of electrically heated homes in the Pacific Northwest. The incremental cost of installing ES windows. Once the evaluation team determined these pivot assumptions, the next step in the analysis was to develop and quantify alternative hypotheses for these assumptions. This task is discussed in the next section. Alternative Hypotheses After identifying the pivot assumptions associated with the ES Windows program accomplishments, the next step is to identify both meaningful alternatives to these assumptions and their likely occurrence (i., their probability distribution). This section presents the values for the evaluation team s alternative hypotheses as well as the source of this information. 31 In the ACE model used for this analysis (CE-C97-028-ES-Windows-MPER5-AAA2001-SH-LC.xls, run date April 11 , 2002), these key savings numbers have a note saying "Origin Unknown." The source for these numbers should be made explicit and their accuracy verified. Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report Electric Heat Saturation The ACE model for this program assumes that 51 % of all houses in the Pacific Northwest (both new and existing) use electricity for space heating. The evaluation team reviewed information supplied to us by the NW Power Planning Council (NWPPC).32 This spreadsheet shows the saturation of electric space heating to be 42%. Therefore, for this pivot assumption, the evaluation team used the 42% as the mean value, and had a distribution that included the 51 % value at the 80% probability level. It is important to note that the NW Power Planning Council has this saturation number declining over time, as the proportion of new homes with electric space heating is lower than existing homes. (Note: the ACE model does not capture this difference between new and existing electric space heating saturation. J The evaluation team alsochanged the distribution of house type within the electric space heat population to reflect the NWPPC numbers, but did not develop distributions for these proportions. Incremental Costs The Alliance assumes that the incremental cost ofES windows is $0.47/square foot of windows. While costs did change owr time as manufacturers applied new technologies more effectively, based on interviews conducted with regional experts and information provided in the MPERs the team judged the average cost to be slightly higher at $0.50/square foot. Because differences in manufacturing costs are not consistent across all manufacturers in the region, the range of values considered was $0.25 to $0.75 per square foot cost differential. Exhibit 4-1 summarizes the above values the evaluation team assumed for the pivot assump tions as well as the value for these assumptions used by the Alliance in their cost-effectiveness model. The distributions for these assumptions can be found in Appendix B. Exhibit 4-1: Pivot Assumptions for ES Windows Assumption Alliance Team Electric Space Heat Saturation 51%42% Incremental Cost $0.4 7/SF $0.50/SF 3 Assessment Findings This section reviews the results of the simulation of the Alliance Cost-Effectiveness model using the above alternative hypotheses on the pivot assumptions. Following the structure of the MAR the evaluation team focuses on the impact of these alternative hypotheses on three main areas: The cumulative savings (aMW) associated with the program. The levelized cost (cents/kWH) from the Alliance and total resource perspectives The cost-effectiveness ratio, also from the Alliance and total resource perspectives. 32 The information currently used by NWPPC on electric heat saturation was supplied to the evaluation team as a spreadsheet, PNWResSectorSupplyCurveUnits.xls. Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report 4-4 To determine these impacts, the evaluation team used the ACE model33 appropriate for this program, and altered the input assumptions as discussed above. Using a Monte Carlo simulation tool, ~Risk, the evaluation team ran 5000 simulations of the ACE model, which pulled observations randomly from the distributions developed for the pivot assumptions. Since this is a retrospective review of the Alliance s performance, the evaluation team discusses only the implication of these alternative hypotheses up to through the venture period, as defined by the Alliance. Cumulative Savings The impact of these alternative scenarios on the cumulative aMW savings through 2002 due to the ES Windows program is presented in Exhibit 4- 2. The change in the electric space heat saturation resulted in a decline in cumulative savings to 2002 from the Alliance estimate of 13. aMW reported in the MAR to an average of 9.8 aMW. Exhibit 4- Cumulative aMW Savings 2002 X 0::=8. 10% X 0::=11. 90% Levelized Cost Exhibit 4- 3 presents the resulting average levelized costs from the Alliance and total resource perspectives for the ES Windows program. The alternative hypotheses increased the levelized cost relative to the Alliance s original results, but the levelized cost in all cases are well below the avoided electricity cost. 33 CE-C97-028-ES-Windows-MPER5-AAA2001-SH-LC.xls, run date April 11 2002. Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report Exhibit 4-3: Average Levelized Cost, ES Windows LEVELIZED COST Alliance Team (cents/kWh) Alliance Perspective Venture + Post Period 0.47 0.46 Venture Period Only 0.38 TR Perspective Venture + Post Period Venture Period Only Cost-Effectiveness Exhibit 4-4 presents the resulting average cost-effectiveness from the Alliance and Total Resource perspectives for ES Windows program. As expected, given the levelized cost results the program is still cost effective from all perspectives during all periods. Exhibit 4-4: Average Cost-Effectiveness , ES Windows COS T EFFECTIVENESS Alliance Team INDEX Alliance Perspective Venture + Post Period 119.83. Venture Period Only 27.19. TR Perspective Venture + Post Period Venture Period Only 2.3 Qualitative Findings The Windows program proved to be a highly successful venture for the Alliance. It was based on the market transformation principles of; elimination of financial disincentives sometimes associated with energy efficient technology, raising the bar on codes and standards for a specific technology, and increasing awarene ss of the benefits of the technology. The Alliance accomplished this by primarily working upstream from the energy end user. Project targets were exceeded ahead of schedule, and the program ended in 2001. Key success factors include; development of cost saving manufacturing processes that did not require retooling; credibility of the implementation contractor among trade allies in the windows market; tailored marketing strategies for individual manufacturers; and retailer incentives. Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report The ACE model associated with the Windows program has seen many modifications over the years, resulting in a model that is not transparent or well-documented. A number of key calculations are performed outside of the model - and those results hard-wired into the current model. While the program is clearly cost-effective, and it is understandable that the Alliance focused on 'getting the program implemented', recommendations were made by the evaluation contractor in March 2000 to update data included in the model. Even so, the evaluation team analysis indicates the levelized costs claimed for the program are within reasonable bounds. Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report ANALYSIS OF BUILDING OPERATOR CERTIFICATION This chapter presents the in-depth evaluation of the Building Operator Certification (BOC) program. SectionS.! provides a brief background on program goals mechanisms, and activities. Section describes the process used to evaluate and bound the impact estimates for the program, and compares the results with Alliance claims. Section presents the findings that emerged from the assessment of the BOC program. 1 Introduction to Program 2 Assessing Program Accomplishments 3 Assessment Findings 1 Introduction to Program Training of building operators in the Northwest began in the 1980s, and the region s first Building Operator Training Certificate program was created in 1993. A number of regional organizations have contributed to the curricula and delivery of BOC program over the years. Beginning in 1996, the Alliance recognized the importance of this work and began funding the endeavor as operated by the NW Energy Efficie~y Council (NEEC) and the Northwest BuildingOperators Association (NWBOA). The Alliance chose to provide major funding for the BOC program from 1997 through 2001. Beginning in 2002, the Alliance chose to provide limited funding for marketing and the development of continuing education curricula.34 This fits well with the market transformation strategy of creating selt:sufficient programs The BOC program is designed to train commercial building operators and facility managers in effective operation and maintenance (O&M) techniques, including optimizing building systems to minimize energy use and increase occupant comfort. While O&M activities have been identified by numerous studies as critical elements of well-run commercial and industrial buildings, building maintenance staff have often historically been ill- informed regarding energy Issues. Currently, two BOC curricula are offered in the region. The NEEC program discussed above has a curriculum it began teaching in Washington in 1996. NEEC licenses its curriculum to partnering entities who deliver the BOC training in 12 other states. In Oregon, the partnering entity is the Northwest Energy Education Institute, which has been delivering the BOC since 1998. NEEC provides certification for students who complete BOC training using its curriculum regardless of which entity implemented the curriculum. NWBOA has a curriculum that it teaches in Idaho and Montana. NWBOA provides certification for the building operators it trains. 34 A more detailed history of the program is available in "Regional Building Operator Certification Venture - MPER #7", Research Into Action, September 2001. Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report The program strategy is to establish a sustainable O&M training environment for Northwest building operators. Program delivery mechanisms are described briefly above, and can be summarized as: Develop curriculum and secure accreditation for training designed to provide the skills required to optimize building performance. Work through several regional organizations to deliver qualified training. Create a model where participants cover the costs of training. Specific progress indicators for the program include: Established and sustained an industry led, voluntary competency based certification process that is recognized and valued by building operators and employees. Secured accreditation and recognition from institutions, employers and facility oriented associations. Increased non-Alliance income; solicited sponsorships and co-marketing opportunities with facility associations, utilities, and large employers. Conducted ongoing market research to identify additional opportunities for certification. Developed and offered two or more levels of training. Total Alliance savings claimed through 2002 for the program are 15.3 aMW, and over 1200 building operators were certified between 1997 and 2001. 2 Assessing Program Accomplishments This section assesses the accomplishments of the BOC program in terms of measurable performance metrics. Based on the MAR, the evaluation team defines these metrics to be the program s electricity savings (in aMW), its cost-effectiveness from the Alliance s perspective and the total resource perspective, and the levelized cost (from both perspectives). As discussed in Section 2., the assessment involves determining the key assumptions underlying the program s assumed accomplishments, developing alternative hypotheses based on these assumptions, and running these scenarios within the Alliance Cost-Effectiveness (ACE) model35 to determine their impact. The sections below present a detailed discussion of these steps. This is followed in Section 5.3 by the results of this assessment. 35 CE-BOC-C97-0250MPER7-MAR-2002-5yr.xls, run date April 2, 2003. Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report 1Pivot Assumptions As was the case for the other programs the evaluation team investigated, the first step in assessing the quantitative impacts of the BOC program is determining the key assumptions required for quantifying the chosen metrics. While there are many assumptions involved in assessing a program, the evaluation team restricted its attention to those inputs that are subject to a relatively high degree of uncertainty or had a significant impact on the outcome of impact estimates. For BOC, the pivot assumptions are: The square footage of facility controlled by each participant. The savings per square foot associated with this training. The lifetime (persistence) of measures installed or actions taken. Once the evaluation team had determined these pivot assumptions, the next step in the analysis was to develop and quantify alternative hypotheses for these assumptions. This task is discussed in the next section. 2. 2Alterna tive Hypotheses After identifying the pivot assumptions associated with the BOC program s accomplishments the next step is to idertify both meaningful alternatives to these assumptions, as well as their likely occurrence (i.e., their probability distribution). This section presents the values for the alternative hypotheses and the source of this information. Overall, the goal was to gather and use the best available information on the identified pivot assumptions. Square Footage Initial program planning data used a value of 50 000 square feet per facility to estimate potential program impacts. Subsequent surveys with building operators and supervisors in the NW put the average facility size an order of magnitude higher. Program evaluators used survey data combined with a 'uniqueness factor' that took into account facilities sending more than one operator for training. This resulted in the ACE model for this program assuming that each participant is responsible for maintaining 234 850 square feet. Additional information gathered during the retrospective evaluation work indicates that this number may be too large, and that the current participant population can reasonably be expected to affect estimated savings for about 000 square feet. 36 Some interview subjects indicated they believed the higher number used for previous estimates was reasonable. Therefore, the evaluation team devebped an alternative assumption that has a mean value halfway between these two values (i., 142,424 square feet). The probability distribution was chosen so that a majority of the values would lie between the two estimates. Savings per Square Foot The Alliance assumed that certification results in a savings of2.5% of a building s electricity use. This implies a savings per square footage number of 0.5 kWh/SF. The study team s research 36 "Education that Changes Behavior: The Impacts of the BOC Program , Marjorie McRae, Jane Peters, Elizabeth Titus, and Tom Rooney, International Energy Program Evaluation Conference Proceedings, August 2003. Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report suggests that certification can be expected to produce a higher level 0 f savings, in the 5-10% range, implying savings as high as 2.0 kWh/SF.37 Therefore, the distribution for this pivot assumption has a mean value of 1.2 kWh/SF, and spans the range from 0.5 kWh/SF to 2. kWh/SF. Measure Lifetime The ACE model for the BOC program assumes that the average lifetime for energy efficiency actions and measures undertaken because of operator certification is 5 years. The evaluation team s review of other certification programs and interviews with instructors suggest that this is probably a good average estimate, but the measure lifetime may be slightly higher for a number of actions taken by participants. Therefore, the evaluation team used an average of 5.7 years. Exhibit 5-1 summarizes the above values the evaluation team assumed fo r the pivot assumptions as well as the value for these assumptions used by the Alliance in their cost-effectiveness model. Exhibit 5.1: Pivot Assumptions , Bee Assumption Alliance Team Square Footage per Participant 234 850 142 424 Savings per Square Foot 5 kWh/SF 2 kWh/SF Measure Lifetime 5 years 7 years 3 Assessment Findings This section reviews the results of the simulation of the Alliance Cost-Effectiveness model using the above alternative hypotheses on the pivot assumptions. Following the structure of the MAR the evaluation team focuses on the impact of these alternative hypotheses on three main areas: The cumulative savings (aMW) associated with the program. The levelized cost (cents/kWH) from the Alliance and total resource perspective. The cost-effectiveness ratio, also from the Alliance and total resource perspective. To determine these impacts, the evaluation team used the ACE model appropriate for this program, and altered the input assumptions as discussed above.38 Using a Monte Carlo simulation tool, ~Risk, the evaluation team ran 5000 simulations of the ACE model which pulled observations randomly from the distributions developed for the pivot assumptions. Since 37 A number of sources were considered, including "Rebuild America" website data that indicate estimated energy savings of 10-20% can be achieved through efficient O&M practices. The Alliance s own Building Performance Systems project overview states 'studies indicate existing commercial building operating performance could be improved from 5 -15% through enhanced O&M practices alone' . One case study presented at an ASHRAE conference in 1997 demonstrated O&M savings of 22.5%. While these numbers reflect significantly larger savings than the BOC program estimated, interviews did not support using that larger number. The evaluation team chose numbers between the conservative estimate of2.5% used by the Alliance and some ofthe cited studies.38 CE-BOC-C97-025-MPER 7-MAR-2002-5yr-Summary.xls with a run date of April 2, 2003 Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report 5-4 this is a retrospective review of the Alliance s performance, the evaluation team discusses only the implication of these alternative hypotheses up to the venture period, as defined by the Alliance. Cumulative Savings The impact of these alternative scenarios on the cumulative to 2002 aMW savings due to the BOC program is presented in Exhibit 5-2. These alternative hypotheses result in a cumulative savings of24 aMW, which is significantly larger than the Alliance s estimate of 15.3 aMW. The Savings at Risk at the 80% quantile is approximately 14 aMW. Exhibit 5- Distribution for 2002 aMW savings/F74 040-. 000 10% 10%5871 36.3973 2Levelized Cost Exhibit 5-3 presents the resulting average levelized costs from the Alliance and Total Resource perspective for the BOC program under alternative scenarios. As was the case in for cumulative savings, the scenarios produce, on average, a significantly lower levelized cost. EXHIBIT 5-3: AVERAGE LEVELIZED COST, BOC LEVELIZED COST Alliance Team (cents/kWh) Alliance Perspective Venture + Post Period Venture Period Only 0.11 TR Perspective Venture + Post Period 0.12 Venture Period Only 0.48 0.40 39 Source: 2002 Market Activities Report, Northwest Energy Efficiency Alliance. Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report 5. 3. 3 Cost-Effectiveness Finally, Exhibit 5-4 presents the resulting average cost-effectiveness from the Alliance and total resource perspective for BOC program. As before, the alternative hypotheses result in a higher cost-effectiveness index for this program than the one developed by the Alliance. Exhibit 5-4: Average Cost-Effectiveness, BOC COS T - EFFECTIVENESS Alliance Team INDEX Alliance Perspective Venture + Post Period 15.1 25. Venture Period Only TR Perspective Venture + Post Period Venture Period Only 4Qualitative Findings Overall, the BOC program is indicative of a program that makes good use of regional resources. The development of curricula and accreditation that is recognized regionally (and to some degree nationally) has value that an independently developed local program would not. Building operators move from one facility to another, and the growing acceptance of BOC certification as a qualification for building staff is a good metric for detennining transfonnation in this marketplace. Targets based on acceptance of certification as a basis for hiring could be used to define an exit strategy for the Alliance. The program leverages its dollars well, and participants now cover most program costs through registration fees. Further market penetration of the training could be achieved through additional co- marketing of training with utilities, professional organizations, and organizations that operate public facilities. While there is a relatively high degree of uncertainty around the energy efficiency actions taken by participants as a result of the training, the cost associated with measuring actual before and after energy intensities at participant and nonparticipant facilities is likely not warranted. 40 Source: Ibid. Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report ANALYSIS OF MAGNADRIVE This chapter presents the MagnaDrive program. MagnaDrive is an entrepreneurial public/private partnership that seeks to accelerate the market acceptance of a new electric motor speed control device. Section 6.1 provides a brief background on program goals, mechanisms, and activities. Section 6.2 describes the process used to evaluate and bound the impact estimates for the program, and compares the results with Alliance claims. Section 6.3 discusses the findings that emerged from this assessment. 1 Introduction to Program 2 Assessing Program Accomplishments 3 Assessment of Findings 1 Introduction to Program MagnaDrive is a startup company, located in Seattle. They have an R&D facility in Port Angeles, Washington. In 1999, the Alliance commissioned a market research study on the speed control market to determine whether there was a viable market niche for the MagnaDrive technology that might expand the current adjustable speed drive (ASD) market. The ASD coupling works by transmitting torque from an electric motor to a load across an air gap. There is no mechanical connection between the shaft and the load, but torque is transmitted via a copper conductor to a magnet rotor assembly and adjusted by varying the gap space between the two. The program strategy is to expand the speed control market, rather than replace variable frequency (VFD) drives. Key applications where MagnaDrive technology may have advantages over VFD are in harsh environments, in vibration sensitive equipment, and in equipment in which motor speeds are generally above 90%. Industries currently targeted by the Alliance/MagnaDrive project are wastewater, HV AC, irrigation, and pulp and paper. The Alliance has invested in MagnaDrive since 1999. Early activities focused on testing and comparing the MagnaDrive technology with other ASD devices. Case studies of early pilot sites were developed and information was provided in various formats to utility industrial reps distributors, and targeted end-use customers. MagnaDrive completed a third round of private equity funding in late 2002. The company also recently had a change in top management. Sales in terms of horsepower installed in the region, while still modest, have grown consistently since 1999. Out-of.region installations now outnumber those in the Northwest. In 2001 , MagnaDrive 41 More detailed program background and review of recent activities can be found in MagnaDrive, Market Progress Evaluation Report, MPER , Quantec, July 2003. Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report installed about 2 HP out of region for every HP in region, and in 2002 about 4 times as much out of region. 42 The program mechanisms can be summarized as: Test and compare MagnaDrive to VFD and other technologies. Demonstrate in- field performance and communicate the benefits to the marketplace. Progress indicators for the program currently include: Determined that the market potential exists beyond the current and projected VFD sales in the NW. Independently verified that the MagnaDrive coupling techno logy saves at least 60% of the energy saved by typical VFDs across a range of speed control applications. Installed the MagnaDrive coupling in at least three field sites where VFDs have been unsuccessful in penetrating the market. Commercialized the MagnaDrive coupling and streamline the manufacturing process. Increased overall awareness of and interest in the MagnaDrive Coupling. Total Alliance savings claimed through 2002 for the program are 1.1 aMW. 2 Assessing Program Accomplishments This section assesses the accomplishments of the MagnaDrive program in terms of measurable performance metrics. Based on the MAR, the evaluation team defines these metrics to be the program s electricity savings (in aMW), its cost-effectiveness from the Alliance s perspective as well as the total resource perspective, and the levelized cost (from both perspectives). As discussed in Section 2., the assessment involves determining the key assumptions underlying the programs stated accomplishments, developing alternative hypotreses based on these assumptions, and running scenarios within the Alliance Cost-Effectiveness (ACE) model43 to determine their impact. The sections below present a detailed discussion of these steps. Pivot Assumptions The MagnaDrive program involves a range of activities associated with developing a start up company based on a manufactured technology, including promoting a specific product to the marketplace. Therefore, there is not a wealth of information available that can be used to develop alternative scenarios. Unit energy savings estimates available were developed by the Alliance, 44 and resources were not available to develop an independent estimate. Sales data are assumed to 42 Sales data provided by Jim Cich, MagnaDrive and Jeff Harris of the Alliance. Out of region sales totaled 6160 HP in 2001 and 21 071 in 2002.43 CE-C99-051-MagnaDrive-MPER1-AAA2001-LC.xls run date Feb. 24, 200244 The savings per HP used for impact calculations are based on performance tests conducted at OSU laboratories and analysis of 6 field site case studies. Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report be reasonably accurate. The variables the evaluation team thus investigated as pivot assumptions are: The horsepower of the installed MagnaDrive units in the Pacific Northwest. The expected market growth of the demand for VSDs in the Pacific Northwest. Once the evaluation team determined these pivot assumptions, the next step in the analysis was to develop and quantify alternative hypotheses for these assumptions. This task is discussed in the next section. Alternative Hypotheses After identifying the pivot assumptions associated with the MagnaDrive program accomplishments, the next step is to identify both meaningful alternatives to these assumptions their likely occurrence (i., their probability distribution). This section presents the values for the alternative hypotheses as well as the source of this information. The discussion focuses on the average value of these alternatives. Appendix B provides information on the distribution about these average values. Installed Horsepower The ACE model for this program does not contain information on actual installations of MagnaDrive units.45 Therefore, the evaluation team determined that this is an important pivot assumption. The evaluation team obtained the actual installation rates for 1999 to 2002 and placed these into the model. The team then assumed that there was no tnlcertainty associated with these figures, and so did not develop a probability distribution for these inputs. In general the total sales of MagnaDrive units during this period were approximately half the amount forecasted in the ACE model. VSD Demand Growth The other pivot assumption in assessing the performance of the MagnaDrive program is the demand growth for variable speed drives (which is directly proportional to the demand growth of MagnaDrive within the ACE model). The ACE model assumes a 5% growth in demand for VSDs. Given that the actual MagnaDrive sales are significantly below the initial forecast, the evaluation team believed that this growth rate was probably toward the high end of the distribution. For the alternative hypothesis, the evaluation team assumed that the average growth rate was closer to 2.5%. Exhibit 6-1 summarizes the above pivot assumptions for the MagnaDrive program 45 The Alliance does receive the actual installed MagnaDrive units (measured in HP), and reports these results in the MAR. However, the ACE model is not updated with these actual installations, and so the levelized costs and cost effectiveness numbers presented in the MAR do not reflect actual historical performance of the program. Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report Exhibit 6-1: Pivot Assumptions, MagnaDrive Assumption Alliance (ACE)Team (and 2002 MAR) Historical Installed HP46 1999 108 553 2000 951 3,469 2001 651 332 2002 462 223 VSD Market Growth 3 Assessment Findings This section reviews the results of the simulation of the Alliance Cost-Effectiveness model using the alternative hypotheses on the pivot assumptions. Following the structure of the MAR, the evaluation team focuses on the impact of these alternative hypotheses on three main areas: The cumulative savings (aMW) associated with the program. The levelized cost (cents/kWh) from the Alliance and total resource perspectives. The cost-effectiveness ratio from the Alliance and total resource perspectives. To detennine these impacts, the evaluation team used the ACE model appropriate for this program, and altered the input assumptions as discussed above. Using ~Risk, the evaluation team ran a Monte Carlo simulation to give a spectrum of possible outcomes. Since this is a retrospective review of the Alliance s perfonnance, only the implication of these alternative hypotheses up to the venture perbd, as defined by the Alliance will be discussed. Cumulative Savings Since the infonnation on installed MagnaDrive units was assumed to be accurate, and the evaluation team did not have the resources to develop meaningful alternative assumptions aboli: the savings associated with each unit, the cumulative savings to 2002 associated with the MagnaDrive program are assumed to be nonstochastic. Thus, the cumulative savings through 2002 are 1.12 aMW, which matches the cumulative savings reported in the 2002 MAR. Levelized Cost Exhibit 6- 2 presents the resulting average levelized costs from the Alliance and total resource perspectives for the MagnaDrive program under the alternative scenarios. The decrease in the 46 The table presents the Alliance predicted installed HP at the time of project adoption, as used in the ACE model to calculate Alliance levelized costs presented in the MAR. These HP values were updated by Alliance staff to reflect actual installations in the Northwest, and the updated aMW are presented in the MAR. The Team installed HP corresponds to these updated values used in the 2002 MAR. Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report 6-4 Exhibit 6-2: Average Levelized Cost, MagnaDrive LEVELIZED COST Alliance Team (cents/kWh) Perspective Venture + Post Period Venture Period Only 2.18 TR Perspective Venture + Post Period 1.28 1.34 Venture Period Only 7.41 growth rate significantly increased the levelized cost of the program, and during the venture period, the levelized cost is above the 4 cents/kWh avoided cost. Cost-Effectiveness Finally, Exhibit 6-3 presents the resulting average cost-effectiveness from the Alliance and total resource perspectives for MagnaDrive program. Given the relatively high levelized cost during the venture period, the cost-effectiveness index from both the Alliance and the total resource perspectives are below 1 during the venture period. Exhibit 6-3: Average Cost-Effectiveness Index, MagnaDrive COST EFFECTIVENESS Alliance Team INDEX Alliance Perspective Venture + Post Period 28.22. Venture Period Only 1.2 TR Perspective Venture + Post Period 1.9 Venture Period Only 0.4 Qualitative Findings The MagnaDrive program is an entrepreneurial program that focuses on commercializing a specific technology. Some regional sources expressed concern that this does not promote the development of a robust market for a product or service. Others felt that these types of programs are an excellent use of regional resources, because individual utilities and organizations would not have the resources to independently test and promote new technologies. Other findings: Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report The MagnaDrive program has successfully demonstrated the technology at a number of sites, and communicated the benefits through case studies. MagnaDrive (the company) would benefit from increased interaction with Alliance staff, and the development of more joint marketing activities. The technology is not appropriate for many industrial applications, where VFDs are still the technology of choice, because of their operating cost benefits at lower speeds. Levelized cost numbers reported in the MAR are based on proj ected values, and were not adjusted to reflect actual aMW savings estimates to date, while the actual aMW savings are reported. While there may be justification for assuming that the projected long-term savings will be achieved, consistency should be maintained for all numbers reported in the MAR. Any time actual savings numbers rather than projected numbers are reported, the ACE model should be re- run to assure internal consistency of the numbers. Interactions with Industrial Customers Some industrial customers feel that Alliance activities to promote technologies such as MagnaDrive overlap with national efforts conducted by DOE, EPRI, and others. Concern was expressed that the program focuses on a specific technology and then looks for appropriate applications rather than focusing on understanding the process needs of a specific industrial sector, and then assisting with the selection of the appropriate technology. Corporate energy managers in the Northwest are relatively sophisticated, so assisting them with a specific technology may not be the most appropriate approach. For smaller industrial companies educating them about the benefits of a new technology may help solve their specific problems. In either case, demonstrating an understanding of the issues facing the customers being approached is critical to their consideration of the technology or service being proposed. The concern discussed above regarding credibility with industrial customers is one that utility industrial representatives constantly face. They sometimes spend years developing credibility with key customers, and get justifiably protective of the relationships they have built. While it may be advantageous for the Alliance to work directly with industrial customers in some cases utility reps would like to always be contacted before calling their key accounts. While the Alliance made concerted efforts to promote the technology to utility industrial program managers, additional coordination with utilities would benefit the MagnaDrive program. CUlTently, some utility ASD programs will not provide incentives for MagnaDrive, but will for VFD technology. Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report SUMMARY ANALYSIS OF OTHER ALLIANCE PROGRAMS As part of the review of Alliance activities, the evaluation team conducted an initial overview of all programs. After the four programs discussed in the preceding chapters were selected for detailed review the team continued to consider the market transfonnation accomplishments of other programs in the Alliance portfolio. A set of programs was selected for summary analysis that would be representative of the remainder of the portfolio, cutting across sectors and implementation mechanisms. The assessment presented in this chapter focuses on describing the lessons learned as they relate to the primary program mechanisms involved which include the following program types - entrepreneurial, consumer, training, and upstream. 1 Program Selection 2 Overview of Successes and Lessons 1 Program Selection In the early years of the Alliance, each new project was presented to the board of directors for evaluation and approval. The board later delegated the detailed evaluation of prospective ventures to a committee, as recommended in the operational audit conducted in 199847 Currently, this Portfolio Committee meets monthly to review both internal and external proposals for new or modified project activities. The criteria used by the committee for selection of new ventures include: financial return electricity savings geographic balance long tenn market impacts customer class reach. Alliance staff prepares Staff Recommendation Memos and Cost Effectiveness Analyses for proposals brought before the portfolio committee. These processes have evolved over time to provide greater detail on costs, potential benefits to the region, timelines, and approach mechanis ms. The portfolio committee then makes a recommendation to the board regarding moving forward with a specific proposal at a designated funding level. In this retrospective evaluation, it was decided to perfonn a detailed review of four programs and an overview assessment of a few additional programs in order to provide a cross sectional review 47 Northwest Energy Efficiency Alliance Operational Audit, Price Waterhouse Coopers, December 1998. Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report of the Alliance portfolio Through a series of discussions with the ad- hoc committee and Alliance staff, the following programs were selected for summary assessment: Sector COM IND IND INFO RES COM IND INFO Mechanism Multi Ie Mechanisms Entre reneurial Consumer Trainin & Consumer stream educational stream Trainin - u stream Trainin - consumer stream 2 Overview of Successes and Lessons This section presents some lessons learned for each of the four program categories - upstream training, entrepreneurial, and consumer (end - use). Upstream Programs Upstream program approaches can be applied to any customer segment. Of the programs singled out for detailed analysis in this evaluation, the Windows program is a classic example of an effective upstream program. While these programs employ a variety of mechanisms, the most common actions applied to upstream programs at the Alliance include: Identification of market barriers, regulatory hurdles, and marketing needs through surveys and other data collection activities. Market intervention through standards development, manufacturer process improvements trade ally interactions, and retailer support. Assistance in promoting and marketing products to create consumer awareness and pull' Support of local utility acquisition programs. For example, since its inception in June 2001 , the Commerical Buildinf!s Initiative (CBI) program employs a variety of mechanisms that had been used by the six fonnerly stand-alone programs now under its umbrella. CBI is based on a strategy of targeting specific market segments (schools, hospitals, grocery stores, etc.) based on a set of well defined selection criteria. These criteria include market size, geographic spread, market readiness, and spillover potential. 48 The key to this strategy is to connect the efficiency message to the values of the respective market segment and to understand the market potential for specific products. The 48 More detail on CBI strategy can be found in Market Research Report, Commercial Building Initiative Target Market Priorities by Schick Consulting and Pacific Energy Associates, September, 2002. Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report approach taken to serve these sectors is based on a foundation of education, training, and technical assistance, as well as working with code officials. Marketing messages are based more on relationship building than on advertising. This approach is a long- tenn one, and it is early to tell whether this ambitious program will achieve its goals. A range of views was expressed by interviewees on this approach. Some commented that they had concerns about a such a broad segment based approach - that the Alliance might do better to stick to more focused, individual actions. Others indicated they believed a 'unifying' approach to both Residential and Commercial markets would strengthen the acceptance of Alliance activities. As a result, it will be important to track key market indicators and assess what actions are taken that can attributed to this initiative in the MPERs. The EnerflV Star Washers program is another example of upstream approach at the Alliance. The strategy here was to increase market share of ES washers in the northwest through an increase in national energy efficiency standards. The program evolved from the original Washwise program in the late 90s to include greater support for retailers in the region as well. In 2001 , the washer program was placed under the Energy Star Home Products umbrella. The Alliance played an active role in the development of the new federal washing machine standards through participation in specifications development activities with the Consortium for Energy Efficiency (CEE) and the Department of Energy (DOE). Both organizations give the Alliance high marks for their input into the process, indicating that the Alliance was 'very proactive ' and 'a good friend to the Energy Star managed programs . The increases in market penetration for ES appliances in general, and washing machines in particular, were more dramatic in the NW than most regions of the country. As the 2002 data indicate in Exhibit 7- the Northwest has easily exceeded the target set for the program of sustaining a 15% market share for energy-efficient clothes washers in the northwest. This data is based on sales through national retailers only. The Alliance has worked closely with independents and regional chains in the northwest to consistently increase the number of participating retailers, and achieve comparable market penetrations for most appliances. Some concern was expressed during the evaluation that there was some overlap between the Alliance program and state appliance rebate programs in the region, and it was suggested that better coordination could occur to avoid duplication of effort. The evaluation team also notes that a review of baseline data used for the cost effectiveness calculations be considered, and that attribution for all savings associated with the 2004 standards could be overstating the Alliance role in the development of national standards. The Alliance plays an active, though background role in the development of codes and standards in the region. Interviewees gave the Alliance high marks for partnering with local, state, and national bodies to improve energy codes across a wide range of products and sectors. In summary, a review of progress indicators for Upstream Programs shows that: The Alliance has done a good job supporting energy code work in the region. Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report Ex h i b i t 7 - Re g i o n a l M a r k e t P e n e t r a t i o n o f E n e r g y S t a r P r o d u c t s Re q i o n Av e r a g e o f Al a s k a ! Lo w e r Mi d - Ne w Ne w So u t h ! So u t h w e s U Up p e r al l Ha w a i i Ca l i f o r n i a Mi d w e s t At l a n t i c E n l a n d Yo r k No r t h w e s t So u t h e a s t Ro c k i e s Mi d w e s t li a n c e s Ai r C o n d i t i o n e r s 35 . 36 . 25 . 37 . 47 . 51 . 40 . 23 . 34 . 38 . 37 . Cl o t h e s W a s h e r s 23 . 4 % 22 . 11 . 15 . 25 . 17 . 31 . 10 . 22 . 16 . 19 . Di s h W a s h e r s 30 . 39 . 37 . 36 . 31 . 33 . 34 . 40 . 4 % 33 . 32 . 35 . Av e r a 28 . 2~ o 3J _ 22 . 32 . 4 0 23 . 28 . 20 . 26 . 4 0 . 28 . 4 o ht t p : / / e s t a r 7 . en e r g y s t a r . go v l i n d e x . cf m ? c = re p s _ re s re t a i l . p t re p s _ re s re t a i l Th i s i s s a l e s d a t a f r o m n a t i o n a l c h a i n s , w h i c h a c c o u n t f o r r o u g h l y 4 0 % o f a l l U . S. r e t a i l s a l e s ( s o u r c e : a b o v e w e b s i t e ) . Su m m i t B l u e C o n s u l t i n g St r a t u s C o n s u l t i n g NW A l l i a n c e R e t r o s p e c t i v e F i n a l R e p o r t 7- 4 The CBI 'segment' approach has improved access to architects and trade associations , but the jury is still out on the overall effectiveness of the new market segment program umbrella approach. Market share of various Energy Star products, including washers, is higher in NW than the country as a whole. Consumer awareness related to energy efficient products has increased significantly in the region over the past several years. Participating Energy Star retailers increased, and range of products available has increased in many instances. Upstream programs pose difficult evaluation problems, but the study team believes that it is important to verify the in- field performance of any technology promoted through and Alliance project. This can be done by working with retailers to track a small sample of customers, and then survey these customers to help ensure that the operating assumptions contained in the CE analyses are, in fact, representative of what is taking place in the field. 7. 2. 2 Training Programs Education and training programs can take many forms to serve market transformation needs. The BOC program reviewed in Chapter 5 of this report provides a clear example of the advantages of regional coordination for a certification process that serves market actors in the commercial sector. The Alliance has engaged i~ a number of other training-based ventures including the Lighting Design Lab (LDL), Compressed Air Challenge (CAC), and Scientific Irrigation Scheduling (SIS). Typical implementation mechanisms for training programs include: Identify market actors and end-users who affect energy use. Develop curriculum and tools to educate these market actors on the benefits of energy efficient practices and products. Increase awareness of the offerings through outreach within respective sectors. Conduct training and measure effects. The Lif!htinf! Desif!n Lab (LDL) project began in 1998 , when the Alliance and Seattle City Light (SCL), who has operated the lab in Seattle since 1989, initiated a collaborative effort to broaden the reach of those that used the facility. Currently, the University of Washington operates a daylighting lab housed at the LDL. Recently, day lighting labs at the University of Oregon were opened and made available for consultations as well. The strategy of this program is to change design practices in the NW to include state-of-the-art day lighting design, and to include high 49 Earlier this year, the Alliance was named one of the Energy Star Award winners by EPA for Leadership in Energy Efficiency. Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report efficiency lighting products in buildings. The labs provide an infrastructure resource for the region s architects, lighting designer, electrical engineers, building operators, and facility managers. The lab provides individual consultations with these building professionals to develop lighting strategies suited to a particular project. In addition to training activities and modeling of lighting designs, a mock-up of a design or equipment option can be physically tested before expenditures are made for an entire building. To date, over 100 daylighting model studies have been completed, and over 2000 individuals received training through the lab's efforts. Savings associated with this program are not tracked, nor were data readily available on the number of facilities actually constructed based on consultations with lab personnel. The program appears to be achieving improved outreach by conducting training at sites throughout the region. Use of the lab resources by architects in the region still has room for improvement, though. Interviewees indicated the lab provides a valuable service, and adds to the regional efficiency infrastructure. The collaborative nature of the activity (Alliance, utility, and university) should prove to induce long- tenn change in lighting design practices in the northwest. Some analysis of actual energy savings achieved through implementation of LDL consultations or training would be valuable. While these savings can be difficult to measure, a process similar to that used to estimate BOC savings could be used. The Compressed Air Challenf!:e program is based on a similar strategy to that employed by BOC to develop an independent training and certification program except that CAC also seeks to build a national collaborative. The program conducts compressed air mamgement courses and has coordinated those courses with state energy offices and utilities in the region. While the total number of individuals trained through the program is not large, Alliance costs have been limited, and industrial participants indicated there was value to the training, particularly associated with the best practices infonnation provided. Savings for the program are not tracked. A separate, but related program that sought to increase the efficiency of pneumatic conveying systems Just Enough Air ended in 2002. It incorporates a service program, along with the training elements, and best practices guide. A number of industrial pilot sites saw significant reductions in costs associated with operating their pneumatic systems, but overall program savings did not outweigh program costs during the venture period. Scientific Irrif!:ation Schedulinf!:seeks to educate irrigators in the region to measure crop moisture needs, and use this infonnation to provide the right amount of water at the right time. A key component of irrigation scheduling is access to and use of accurate weather data. The Agrimet Weather Station Network has assisted in achieving this. The Soil Moisture Data Logger provides another tool for irrigators to help detennine optimum irrigation schedules. This project ended in 2000. At that time, evaluation interviews with outreach coordinators suggested that SIS will become common practice for 50% of growers who irrigate within 10 year, but that Alliance efforts may be too short-lived to have the effect initially targeted. The evaluation team did not interview agricultural market actors in the northwest. In summary, the status of Progress Indicators for training programs indicate: Case study development was successful in many instances Market awareness increases evident through adoption of new practices in both C&I sectors Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report There has been some progress on creating selt:sustaining entities to deliver training Regional energy-efficiency educational infrastructure has improved based on Alliance activities. One interviewee commented "these education programs have a high value to the region It would be useful to track participants in educational programs and conduct short surveys to determine what actions are actually taking place in the field and whether the individuals attribute some aspect of these actions to the training received. 3Entrepreneurial Programs The Alliance s entrepreneurial ventures seek to help new, specific efficiency technologies and services find market acceptance by supporting the efforts of start-up enterprises. The mechanisms chosen to achieve this strategy include: Identify fledgling tools or processes that will improve energy efficiency within a given sector or end use, and examine whether market potential exists. Independently test the technology to assure it works as claimed. Identify regulatory or other hurdles that may limit product success. Assist in developing realistic business plans to bring the tool to market. Provide ongoing assistance to assist entrepreneur in reaching market. The MagnaDrive program was discussed in some detail in Chapter 6 of this report. BacGen another example of an entrepreneurial program that the Alliance has supported. BacGen Technologies approached the Alliance in 1997 with a proposal to save energy by working with wastewater treatment facilities in the region to increase the efficiency of their operations. The approach focuses on a reduction of mechanical aeration at wastewater facilities, and the company provides a range of diagnostic and implementation services to the wastewater industry. The 1\lliance has assisted with business plan development, pilot site implementation & funding, case studies and funding coordination. BP A has provided additional funding for BacGen installations through its efficiency resource acquisition program. Currently, BacGen has about 20 sites in the northwest, and about 100 in California. There are remaining market barriers, including lack of understanding of the technology and approach among state health regulators, and a lack of capital for BacGen activities among targeted customers. The Alliance s local government reps help to arrange meetings to address the regulatory barriers, and some facilities are entering into ESCO agreements with BacGen to address the lack of capital concern. The Alliance has recently reduced its financial role to essentially providing only marketing dollars. Some interviewees expressed concern with the entrepreneurial approach, indicating their belief that by supporting one company and a specific technology, the Alliance does not necessarily help Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report create a robust, competitive market for a set of products or services. The approach is quite different than those employed for upstream and training programs, and the Alliance will need to address these concerns going forward. Alternatively, others interviewed indicated that for utilities, having the Alliance do the vetting of potential new energy efficiency technologies was an excellent use of a regional resource - as it reduces duplication of effort by regional utilities and allows for investigation of technologies that an individual utility would not have the resources to test. While this is certainly true, industrial customers are sometimes best served in this regard through national efforts underway for specific targeted industries, and care should be taken to involve a wide range of stakeholders before launching new industrial focused entrepreneurial efforts. In terms of implementation, care in conducting thorough due diligence before making an investment in any start- up company must be assured. Picking a promising energy efficiency technology is only the start. Assessing the target market niche, analyzing competitive technologies and companies, and deterrpining whether regulatory or other barriers can be overcome in a timely manner through cooperation between the Alliance and the start- management team all come into play in the process. The views expressed by stakeholders and market actors indicate that while the Alliance has been thorough in some of these activities, the due diligence has not always been consistent. While a private venture capital firm may choose to apply different levels of analysis to potential investments, the investments made by the Alliance involve the use of stakeholder funds, and will continue to be scrutinized closely. In summary, the status of Progress Indicators for entrepreneurial programs indicate that: Technologies have been successfully tested and demonstrated. To date, there has been limited market acceptance of program technologies and processes. Evaluators commented on the BacGen program that "evidence for widespread market transformation has not significantly surfaced. The Alliance must continue to assess the market potential of new technologies, develop case studies, and promote benefits to the right market actors. 4Consumer (End-use) Programs Consumer programs are characterized by activities that are targeted directly to energy end users and can be focused on any customer class. Key mechanisms for these programs include: Raise awareness regarding benefits of an efficiency product or process. Demonstrate the benefits to the respective end-user group. Establish local support for education and technical assistance. While some of the training programs discussed above are targeted directly to end- users, the primary mechanism to induce market change in those is through education of the end-user or trade ally, not the implementation of a specific technology. Consumer programs have historically been the focus of utility resource conservation efforts, and some parties interviewed Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report indicated a belief that is where these programs should reside. Others expressed the need for the Alliance to target specific regional industries with technology programs that may be more difficult to develop at individual utilities. This is the case with the Evaporator Fan VFD Initiative which began in 1998. This project's strategy is to make VFDs a standard technology for evaporator fans in NW IDod storage facilities. They employ an implementation contractor in the region that has significant experience with industrial refrigeration systems. The program has focused on demonstrating to owners and operators of refrigerated warehouses that the technology has financial benefits, and is reliable. It has proven to be critical demonstrate that the technology has at least a neutral impact on fruit quality, and to disseminate this infonnation to the broader marketplace. Demonstration at approximately 20 facilities showed fan energy savings of 24- 78%, and mass loss of stored fruit improved slightly as well. The acceptance of the technology within the target market sector is on the upswing, and the most recent evaluation suggested that "VFDs will become ubiquitous over the next 1 0 years, with or without utility programs . The program has met its goals of conducting field trials, and developing a database of storage facilities.. It has long tenn (2007) targets of increasing market penetration of VFDs to at least 31 % of warehouses and 47% of fruit storage facilities. Evaluators also recommended developing an Evap Fan VFD consultant base, in order to develop a more robust market for services than now exists, and to provide an array of infonnation resources - including expanded case study distribution to market actors, a VFD application guide, and web resources. The status of Progress Indicators for consumer programs can be summarized as follows: Savings are not tracked for a number of these programs, but the study team believes that it would be possible for some of these programs to perfonn case studies and review field perfonnance to see if the technologies that are being supported by Alliance efforts are having the anticipated impacts. Achievements are mixed, solid progress has been achieved on a number of fronts while financial successes are less clear. Coordination with local utility programs and QC procedures for implementation contractors who interact with utility key account customers is critical to future success. Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report OVERALL VALUE ASSESSMENT This chapter provides a summary of the findings from the scenario analysis conducted for the four programs assessed in detail, and the effects on the overall cost effectiveness of the Alliance. It also presents qualitative findings regarding Alliance activities that are not program specific. Section 8.1 presents the distributions and overall energy savings associated with Alliance activities, and summarizes the program impacts. Section 8.2 summarizes the overall value of the Alliance based on the team s interviews with stakeholders and market actors in the region. 1 Estimated Impacts of the Alliance - Energy Savings and Levelized Costs 2 Overall Value of the Alliance 1 Estimated Impacts of the Alliance Energy Savings and Levelized Costs Chapters 3 through 6 addressed the impacts of four selected programs based on a detailed analysis by the study team and a tracing of the numbers through the cost-effectiveness models with the addition of new information (when appropriate) from other studies and interviews conducted. In addition, this process used ranges instead of point estimates for impacts to dimension the uncertainty in some of these estimates. This section takes the results from these individual program analysis efforts and examines their impact on the overall energy savings claimed by the alliance. A key question associated with the analysis is how the results of the investigation into alternative scenarios for the four programs affect the overall Alliance investment perspective. To address this issue, the levelized cost from the Alliance perspective through 2002 was used. The calculation used for this is: Total $ spent to date on all ~grams Levelized Savings to Date Using the Alliance numbers from the 2002 MAR for the costs and savings from all programs produces a levelized cost of 0.7 cents/kWh. The study team s analysis of the savings associated with the ES Lighting, the ES Windows BOC, and MagnaDrive programs, when combined with the Alliance claimed savings for other programs, produces an average cumulative savings to date of 98 aMW for all programs - using the low influence scenario for the CFL program (compared to the Alliance s original estimate of 134 aMW). The distribution for the total savings to date is presented in Exhibit 8- Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report Exhibit 8- Distribution for Overall Cumulative Savings Through 2002 Distribution for Annual Savings aMW 2002/E30 0401 000 Mean=97.57 438 100 125 150 1 00/0 80.9308 113.1527 10010 Note: Savings shown use team estimates for 4 programs reviewed in detail, including the low influence scenario for the CFL program. Using this distribution for total savings through 2002, in the above levelized cost equation, the probability distribution for this levelized cost shown in Exhibit 8- 2 is obtained. Exhibit 8- Distribution for Levelized Cost, Alliance Venture Period Perspective Distribution for Levelized Cost / PV Costs/Benefits 2002... 350-. 75 11. Values in 101\~O% 3745 11.7073 14. 10% Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report The revised analyses documented in Chapters 3 through 6, when combined with Alliance estimates for other programs, result in a study team totallevelized cost for all the Alliance activities (from the Alliance Venture period perspective) of 0.99 cents/kWh, with a 90% probability that the cost is 1.17 or less, and a 10% probability that the cost is less than 83 cents/kWh. While this is an increase in the estimated levelized cost, it is still well below the avoided cost of power in the region. When levelized costs are considered from the TRC 2010 perspective, the team s estimate of levelized cost for the entire portfolio is between 13 cents/kWh and 1.29 cents/kWh, with a mean value of 1.21 cents/kWh. The composite changes across the four programs analyzed produce the new venture period levelized cost estimates shown in Exhibit 8-3 below. These estimates represent the best available information that was available to the study team. While there are several reasons for the differences between the results developed by the study team and the Alliance estimates, the two key areas of difference are the assumed baseline for CFL sales (e., how many CFLs would have been sold in the absence of Alliance efforts) and in estimates of the savings per unit or application (e., savings per lamp or savings attributable to each person participating in the BOC training). Exhibit 8-3: Average TRC Venture Period Levelized Cost TRC LEVELIZED COST Alliance Team (cents/kWh) Energy Star Lighting - High Scenario .44 1.89 Energy Star Lighting - Low Scenario .44 Energy Star Windows BOC .48 .40 MagnaDrive 7.41 Exhibit 8-4 shows a summary of the venture period benefit-cost ratios (from the Alliance perspective) as determined by the study team analysis, compared to the same ratios developed by the Alliance in the ACE models for each program. Other than the MagnaDrive program, all programs analyzed by the team were cost-effective. 2 Overall Value of the Alliance The interviews conducted with trade allies and market actors contained a set of questions that addressed the overall value of tre Alliance. Questions addressed four areas of Alliance activities - Planning, Implementation, Evaluation, and Communication. These questions were not asked 50 It should be noted that the study team s analysis looked at the four programs from a retrospective perspective. Other than making an adjustment to the future consumer replacement cost of CFLs, no other future costs, or estimated future savings were modified. Neither did the study team analyze local utility costs, or consumer O&M costs related to the programs. The cost estimates used by the Alliance for regional costs other than their own could have an effect on the levelized cost from a TRC perspective, and should be reviewed periodically. Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report Exhibit 8-4: Average Alliance Venture Period Cost-Effectiveness (Benefits/Costs) ALLIANCE COST EFFECTIVENESS Alliance Team INDEX Energy Star Lighting - High Scenario 17. Energy Star Lighting - Low Scenario 17. Energy Star Windows 27.19. BOC MagnaDrive 0.4 of Alliance staff. The individuals interviewed are shown in Appendix A to this report. Theyinclude project implementers, project evaluators, utility program managers, retailers/trade allies and other stakeholders. The respo~ses to these questions regarding the perceived value of the Alliance were combined with other information obtained from the review of reports, and interviews with Alliance staff. This process produced insights that the study team believes are useful for this assessment. Not all individuals interviewed were able to address each of the topic areas; however, when putting these responses and comments in context, there were three areas that were generally addressed by the interviews. Overall Value: Impact of Alliance on Market Transformation The interviews with stakeholders (excluding Alliance staff) indicated: A widely held belief that the Alliance is responsible for higher levels of market transformation than would have occurred had the Alliance not existed. That the Alliance has had a significant impact on a number of markets for energy efficiency technology and their efforts on the energy efficient windows market in the Northwest is exemplary of a successful MT program. That the focus of the Alliance s market transformation efforts has been appropriate, i. the Alliance has done a good job of identifying and pursuing programs best addressed regionally. Also, the study team s reviews of the selection process and the program implemented supported this finding. Examples include training programs such as BOC and projects targeted at manufacturers (e., Energy Star windows) are the types of programs whose implementation spans utility service territories and even state boundaries. The Alliance has developed an infrastructure of programs, facilities, relationships, and personnel that represents an asset to the region. Physical examples of this infrastructure include the Lighting Design Lab and web-based information resources. The relationships developed by the Alliance staff and contractors with local governments, manufacturers and retailers represent another type of capital that provides opportunities for future efficiency activities that would not otherwise exist. Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report 8-4 While the comments above represent the study team s findings based upon the interviews, there were other comments made that did rot appear to be part of the majority view, but the study team believed that they were worth bringing to the attention of the Alliance and its Board for their consideration. Two such comments were: There was one dissenting opinion regarding the impacts of the Alliance on market transformation which was based upon the belief that baselines are dramatically understated , many market changes observed would have happened without the Alliance - due to other factors. Another individual expressed the opinion that while venture selection has been good overall there was concern expressed about the recent selection process. Overall Value: Alliance Tracking of Iml!!!:E1l There was a general view among the stakeholders interviewed that impact estimates tended to be modestly high. Comments pertaining to this finding include: Several individuals expressed concerns about the Alliance claimed impacts for its efforts in residential lighting and this did show up in the bi- modal estimates of CFL sales influence used in the study team analysis in Chapter 3. The individuals interviewed indicated that while they generally believe the Alliance impact estimates were somewhat high, it was believed to be only a modest overstatement on the order of 10 percent or so on average. There was a perceived need to update baselines used to estimate program impacts more frequently. There is a need to test performance assumptions in the field rather than assume that actual performance matches predicted performance. In addition to the sets of comments that are viewed as study team findings above, other comments that may be of interest to the Alliance and its Board are: Evaluators did not feel pressured by Alliance study sponsors to produce favorable numbers or results. Some of the Board memb ers interviewed (and also other stakeholders) indicated that they were "trusting" a few knowledgeable "other" Board members to ensure unbiased impact evaluations since they had expertise in this area. Originally the MPERs were designed to look at entire markets, not just at project indicators and effects; but, over time, they have changed to focus on project effects and delivery processes and less on the overall market. Overall Value: Organizational Effectiveness A question was asked regarding how well the Alliance operated as a business organization. This was asked in the context of overall Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Reporl operational effectiveness compared to other organizations with which the individual being interviewed was familiar. Comments on this topic included: Only one interviewee rated the Alliance as "below average" as a business organization other respondents rated the alliance as well above average (a 7 or 8 on a scale of 1 to 10) and based on other organizational studies, this is a high rating and reflects favorably on how the Alliance operates. Planning was viewed as a strength of the Alliance. Communication to stakeholders was rated as very good. Importantly, alliance staff was well respected by all stakeholders interviewed. Some other comments for consideration tlRt were not viewed as findings are: Some concerns about recent trends in various areas (venture selection, accountability, and transparent accounting), but this was viewed as quite recent and not yet of great significance. The Alliance has faced recent cha llenges due to organizational changes in general, staff turn over and staffing patterns in particular. The Alliance s reputation among trade allies was raised as a "potential" concern - related to this concern was implementation contractor selection and treir ability to relate to the concerns of trade allies and industrial customers in particular, and the implementation of an overall quality control process managed by the Alliance rather than its contractors to ensure that relations with key trade allies remained favorable. Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report ISSUES FINDINGS AND RE COM MEN D A T ION S This chapter discusses the fmdings and recommendations of the retrospective assessment. It is divided into several sections with the first presenting general findings that the study team has classified as "over-arching themes" resulting from the assessment. Section 9.2 discusses aspects of three key processes that are linked; the Cost- Effectiveness (CE) Analyses, the Market Progress and Evaluation Reports (MPERs), and the production of the annual Market Activity Report (MAR). The final section presents the study team s recommendations for consideration by the Alliance and its Board. 1 Over-Arching Themes from this Assessment 2 Issues Related to Cost- Effectiveness and Evaluation Processes 3 Recommendations 9.4 Final Comments 1 Over-Arch i ng Themes from th is Assessment This work effort was comprised of numerous tasks ranging from review of numerous alliance reports and documents, interviews with regional experts and market actors, and quantitative assessment of program accomplishments. It is important in a project of this type to examine how this information might be organized in a manner that can provide a relevant overview of the Alliance. In preparation for the final presentation to the Ad Hoc Retrospective Assessment Committee, the study team reviewed the information from the assessment and developed seven over-arching tl~mes that represent the high level findings. This are discussed below and in greater detail in the subsequent sections of this chapter. THEME 1 The Alliance business culture is characterized by open communications, a focus on the planning and delivery of programs, and no discern able bias. The Alliance has developed a culture of adaptive management and continuous learning that has been communicated throughout the organization. Alliance personnel were open and direct in its communications with the study team. The project interviews with evaluators and implementers indicated that the Alliance worked with them to reach appropriate answers. There were no pressures to produce any specific result or finding, and the general attitude was one of working with tre study team to reach appropriate answers and conclusions, and to continue to make the Alliance a "learning" organization. As a relatively new organization that had the challenge of initiating market transformation programs, the focus of the Alliance to date has been on the planning and efficient delivery of appropriate programs. THEME 2: The Alliance has been successful at transforming, or contributing to the transformation of markets. The study team reviewed Alliance program evaluations, and interviewed the researchers who conducted program specific evaluations, as well as other Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report regional stakeholders and market actors involved in energy efficient markets in the northwest. Based on the information gathered, the study team determined that the Allia~e made substantive contributions to transforming regional markets for energy efficiency equipment and practices. Specifically, the MPERs addressing the Energy Star Windows program were compelling in their documentation of Alliance influence on the market share of efficient windows installed and the increase in the number of active manufacturers of high efficiency windows in the region. Both the MPERs and national data indicate market penetration of Energy Star windows in the Northwest are more than twce the national average. Interviews with industry experts also supported the contention that Alliance activities have permanently impacted the windows market. Other markets where there was considerable evidence of market transformation included clothes washers and CFLs. For clothes washers, the market penetration of Energy Star washers in the NW are significantly higher than the national average (see chapter 7). The market for CFLs changed more dramatically in the NW than other regions of the country, even when the West Coast energy crisis is considered (see chapter 3). The specific magnitude of the impact of the Alliance on these markets was a subject that not all market actors in the region agreed upon, but there was a more general agreement that permarent changes had taken place in these markets and these changes were at least partially the result of Alliance activities. THEME 3 Market Progress Evaluation Reports tended to focus on the program delivery process and on providing feedback for program de sign and implementation improvements. This met an immediate need for Alliance personnel responsible for program implementation, and these reports improved the delivery and implementation process. This trend towards process analyses and providing feedback for program improvement fits well with the Alliance s goal of adaptive management, but the estimates of impacts and the ability to substantiate claims of Alliance- induced market effects would have been enhanced by having the MPERs more directly address savings per unit and the issue of attribution within each study. In general, the MPERs were not structured to provide information on attribution or savings per unit for energy efficiency equipment (or applications) influenced by Alliance activities. THEME 4: Cost-effectiveness analyses were difficult to replicate and the current processes used are cumbersome. Cost-effectiveness models and analysis efforts form the basis for projecting market impacts from programs under development by the Alliance and they are used as the basis for the Alliance claims of cost effectiveness as presented to the public in the MAR. The initial pressures on the Alliance have been to develop and implement programs in the field. The study team commends this past focus, but is errphasizing the need for improvement in this area going forward. The magnitude of claims made by the Alliance as its portfolio of programs has grown will result in new challenges related to accountability that may not have been present (or needed) during tre start- up phase. As a result, a more streamlined and transparent process for estimating and modeling program cost-effectiveness is needed. In general, it is can be easy for those not directly working with a model- based system to underestimate the comp lexities involved in the modeling effort. There is often a question of how to allocate resources within an organization and some tradeoffs are generally made regarding the activities performed. The initial pressures on the Alliance have been to develop and implement Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report programs in the field. The study team commends this past focus, but is emphasizing the need for improvement in this area going forward. The magnitude of claims made by the Alliance as its portfolio of programs has grown will result in new clRllenges related to accountability that may not have been present (or needed) during the start- up phase. As a result, a more streamlined and transparent process for estimating and modeling program cost-effectiveness is needed. The study team traced Alliance claims of energy savings as presented in the MAR and developed alternative scenarios, which were subsequently run through the CE models. These analyses resulted in lower estimates of Alliance impact claims as compared to the numbers reported in the 2002 MAR (from 134aMW to 98aMW).sl Most of this adjustment came in one program-the ENERGY ST AR(8) Residential Lighting program - with adjustments in other programs examined being much less significant. The numbers used in this adjustment are from the "low influence scenario" for the CFL program (see Chapter 3). A comparison of this analysis is shown in Exhibit 9-1. Note that ACE calculated savings were not available for all programs not reviewed, so there is no ACE value shown for 'all programs Exhibit 9- Comparison of Energy Savings Through 2002 2002 Cumulative Savings 160 140 120 100 I/) 80 s:: UJ rs ACE Ii MAR 0 Team ~(f ;-I ~ . Jf ~ ~rp Program 5 I See footnote #26. Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report THEME 5 : Benefits of the Alliance have exceeded costs. Even with the study team adjustment to estimated Alliance energy savings estimates, the analysis of program impacts shows the benefits from Alliance activities have exceeded its costs. In fact, the team analysis indicates a levelized cost for the Alliance portfolio of programs of between 0.83 and 1.17 cents per kWh, with a mean of 0.99 cents/kWh, when they are considered from the Alliance venture period perspective (see chapter 8). THEME 6 The regional approach of the Alliance is an asset and even greater leverage in program implementation can be gained in the future. The Alliance has developed an infrastructure of programs, relationships, and personnel that represents organizational capital that will be valuable in the future. Interviews with market actors indicated some diverse opinions regarding past efforts of the Alliance and the amount of energy savings that should be attributed to alliance activities. Even taking those comments into account, there was a general consensus that the Alliance was able to undertake certain programmatic activities more efficiently on a regional basis than was possible through local efforts. The study team s review of programs selected for implementation by the Alliance indicated that most programs were well suited to implementation by a regional organization. In this respect, the Alliance was living up to its goal of focusing on market transfonnation projects that can best be addressed at a regional level. In summary, the six themes that came out of the study team s assessment work effort show that the Alliance has been an effective organization that has focused on the selection, design and delivery of programs. As a new entity, this was an appropriate focus. With programs that have grown, the Alliance infrastructure in tenns of analysis capabilities needs to grow as well. Issues Related to Cost-Effectiveness and Evaluation Processes One of the key charges that was given to the study team was to examine Alliance claims and work to validate the fact the energy savings attributable to the Alliance justify the costs of the Alliance. To accomplish this objective, the study team begLn with the 2002 Market Activities Report (MAR) which documents the Alliance s estimates of impacts. As discussed earlier in this report, four programs were selected for detailed analysis by the study team. This involved a review of underlying assumption; and hands-on work with the Alliance cost-effectiveness models used to produce estimated savings, benefit-cost ratios, and estimates of levelized costs. This section discusses three processes that work together to produce the Alliance estimates of energy savings and benefit-cost analyses. These three processes are - 1) the cost-effectiveness model(s) used by the Alliance, 2) the MPERs that evaluate projects, and 3) the MAR which consolidates infonnation from these two sources in an annual report. The Alliance Cost Effectiveness Model The study team conducted a review of the Alliance Cost Effectiveness (ACE) model as part of the assessment of the Alliance s claims, particularly for the models used for the four programs investigated in detail. Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report 9-4 The ACE model was originally developed to standardize the analysis across all programs. As result each ACE spreadsheet contains the same structure including worksheets, location of cells and formatting, with program specific calculations are contained on either the "Non-standard Calcs" sheet or the "Input Assumptions" sheet. This makes the consistency across program model analysis somewhat superficial in that many of the meaningful calculations are performed in these non-standard calculation sheets. Given the need to develop different project scenarios some project modeling efforts have developed numerous calculation sections within the "Non- standard Calc" sheet which results in complicated models. One of the reasons for this may simply be time constraints. As in all work environments, Alliance staff have deadlines and time pressures and it is easy to understand why a quick fix might be taken to meet a near term deadline. At some point, it becomes appropriate to "clean- up " and streamline the modeling process even though each individual decision that has been taken in the past may have been reasonable and served the need for practicality and expediency. The difficulty encountered by the study team in tracing CE analysis efforts and identifying which assumptio ns were used points to a need to streamline the CE modeling process. In addition, the current ACE models include the past assumptions used in the model even though these may be outdated and not used in the current calculations. While there are clearly benefits to this approach, it does make it difficult to easily determine which assumptions are used in the model. It might be better to store these past assumptions in a separate document. And finally, to help clarify the model, the team recommends that the necessary input assumptions for the model should be clearly stated on the "Input Assumption" sheet, and no inputs should have to be entered on any other worksheet. A final issue is the application of the ACE model and its relationship to the numbers reported in the MAR, is to strive for consistency across all programs. As an example, ACE modeling of the lighting program was updated to incorporate actual sales of bulbs through 2001, but not through 2002. Alliance cost-effectiveness and levelized cost calculations are thus based on partially updated numbers. Conversely, for MagnaDrive, the numbers in the ACE model are the original planning numbers, even though the Alliance had data on actual MagnaDrive installations available. This may have been a resource allocation decision, i.e., when is it appropriate to update the analysis if the program is still in its early stages. Judgment is always required in making resource decisions regarding when to update impact estimates particularly for programs that are showing relatively small impacts. In general, however, as the Alliance continues to track and report attained impacts, the ACE model should represent the best currently available numbers rather than default back to initial planning values. This isSle is made somewhat more complex by the fact that the Market Activities Report uses the actual accomplishments (in terms of aMW) to date for each program, while standard practice it to not change these assumptions in the ACE model. This can result in subtle differences between the cumulative savings reported in the MAR and those used to calculate the levelized costs and cost effectiveness indices also reported in the MAR. Updating the ACE model to use the actual accomplishments would eliminate this incon;istency. 52 For example, in the ACE model for windows (CE-C97-028-FS-Windows-MPER5-AAA200I-SH-LC.xls, run date April 11 , 2002) some inputs are found on the "Input Assumption " worksheet while other inputs are found on the Non-standard Cales" worksheet. Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report The study team does not believe that the issues raised above in the application of the ACE models and the incorporation of the results in the MAR has led to substantive miss-reporting of results. However, as programs grow and attribution of impacts to Alliance activities continues these issues could become important in the future and should be addressed on a going- forward basis (see recommendations). It is the study team s opinion that, at this point in the development of the Alliance, some additional effort to organize, document modeling efforts, and streamline the ACE process will produce benefits for the Alliance. 2Market Progress Evaluation Reports In developing the review of the Alliance s accomplishments, the team reviewed a large number of the available MPERs. Since the retrospective objective as to determine impacts, i., to what extent has the Alliance contributed to energy savings and moved markets, the study team review concentrated upon how the MPERscan be used to determine the program progress. As a general statement, the study team s review of the MPERs indicated that most MPERs reviewed market indicators and provided some input into important indicators such as market share calculations. Expected savings per application associated with the project and related assumptions used in the cost-effectiveness model were not researched or not researched at the same level of detail. Essentially, the MPERs focused primarily on assisting the Alliance on project implementation feedback, with some verification of market effects and a limited review of the input assumptions to the ACE model. The study team believes that a modest incremental effort would be required to more explicitly explore project impacts (through a more careful assessment of input assumptions) and attribution of impacts to Alliance activities. In many instances, this could be accomplished by adding a few questions to surveys that were fielded by the evaluation contractors. Overall, the MPERs are a good process review of program delivery, but did not research project savings as actively as might have been expected. Balancing research objectives and allocation of resources is generally an issue and interviews with individuals performing the project evaluations indicated that they believed the focus of the work was on analysis of the program design and delivery as part of the Alliance s adaptive management process. This may be appropriate, but the study team believes that the overall objectives of the Alliance can be better met by providing greater direction to the project evaluators regarding the need to estimate attribution, in- field performance of technologies, and assumptions used in the cost-effectiveness models. Ensuring that expected energy savings from Alliance projects are indeed taking place in the field is also an important objective as the Alliance portfolio of programs grow and become more diverse. 53 There were some instances where the MPERs did provide guidance on input assumptions and these often were incorporated into the cost-effectiveness analyses. But, this did not always occur. In one of the four programs investigated - energy-efficient windows -- the project MPER made recommendations regarding the cost- effectiveness calculations. MPER #3, Quantec, March 2000 states that "Some of the issues raised in this report are due to 'age' of data utilized by the Alliance." In particular, estimates of housing starts, fuel mix, and energy savings had not been updated since 1995(or earlier in the case of fuel mix), when they were used to develop the Fourth Northwest Conservation and Electric Power Plan. More recent studies (e., a study by Ecotope) showed different electric heat saturations than those assumed in the ACE model. While the current ACE model for this program does contain a detailed section containing the Ecotype numbers, this section of the model was not used to compute the final results. At the time of this report, the cumulative savings, levelized costs, and cost-effectiveness index computed by this ACE model is still based only upon the overall electric space heat saturation from the Fourth Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report 3The Market Activities Report The need for accuracy in the CE analyses and the role of the MPERs depends in part on what the Alliance and its Board hopes to accomplish with the energy savings estimates that are attributed to the Alliance in its MAR. The MAR is widely viewed as the 1\lliance s defmitive statement on what it believes it has accomplished. The language used in the MAR supports this assumption. For example in the recent 2002 MAR, it states that For the reporting period January 2002 to December 30, 2002, the Alliance has saved 45aMW, as depicted in Chart In the next paragraph of the MAA Executive Summary it states that Total Savings of the Alliance since its inception in 1997 (exclusive of utility direct rebates) 134aMW. It was not uncommon in interviews for the respondent to directly quote these numbers and wonder what support they had, i., were they conservative and savings might be even higher or were they optimistic. In one case, the respondent referenced the MAR and said "1 know that the Alliance didn't save 134aMW, 1 just don t know whether the real number is above or below that value." The implication of this statement, in the opinion of the study team, is that these relatively absolute statements regarding Alliance accomplishments may be over-reaching and for at least some stakeholders, it has the unintended effect of raising some concerns about the balance and credibility of the Alliance. While most stakeholders interviewed indicated they felt the Alliance staff takes great care to make these numbers accurate, it is important to note the dissenting view. The MAR has high quality graphics and is authoritative in appearance. The Alliance energy savings claims are very direct and unequivocal. Given this, what is the appropriate underpinning and evidence needed to ensure that these claims are viewed as credible by entities that work with and support the Alliance? One option would be for these estimates to be given in terms of ranges and scenarios could be presented that represent "conservative" estimates. This also translates into a question concerning what should the MPERs be reporting in terms of program impacts and pivot assumptions? The annual MAR addresses market activities, but it also measures program progress in terms of cumulative savings achieved, levelized costs, and cost-effectiveness indexes. The MPERs could, with little additional cost, provide better foundations for estimates of energy savings by making sure that the evaluator is aware that they have to verify current project assumptions used in the CE analysis, and they have to develop estimates of attribution to Alliance activities based on the best available information, even if this is comprised of expert judgment developed during the course of the evaluation. The study team believes that it is important that the Alliance provide estimates of energy savings accomplishments, but there may be better approaches to bracketing and bounding the estimates Northwest Conservation and Electric Power Plan. This reflects on the current complexity of the documentation and CE models in current use, and the fact that it is sometimes difficult to determine which assumptions were actually being used in the calculations the produced the model outputs. Finally, if a program is believed to be clearly cost- effective, as the windows program was, is it worth the effort to update assumptions and re-run the model. Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report than are currently used in the MAR. Even with that change, the MPERs should refocus and provide additional support for energy savings estimates. Recommendations A number of specific recommendations are made by the study team in this section. These recommendations reflect the issues discussed in the preceding sections. The recommendations are divided into five categories and are presented below. Recommendation Area #1 Cost- Effectiveness Models and Processes: RI.I - The Cost-Effectiveness spread-sheet models should be "cleaned-up" and a better documentation process implemented to avoid confusion in the future. The study team believes that several man-weeks devoted to this process could considerably enhance the transparency, and user-friendliness of these important tools and save labor hours down the road that might well make up for the short tenn costs of upgrading the CE process. 54 Rl.2 - Develop more specific processes to update and track assumptions used with direct links to sources of assumptions and referencing the MPERs that are tasked with reviewing the input assumptions for each project. RI.3 - The Board needs to recognize the complexity of this work element and help ensure that there is a reasonable review process for model results. The fact that a model is used does not, in itself, ensure that the outputs are appropriate. Recommendation Area #2 Use of scenario analysis and identification of pivot factors in reporting of Alliance accomplishments: R2.1 -Evaluation and planning (e., the venture business plan) would benefit from the use of bounding scenario analyses and the identification of pivot factors. The portfolio committee discusses these fa ctors when deciding whether a specific project should be undertaken, but these assumptions need to be documented and tracked over time. This would provide the following benefits: Large uncertainties on select pivot factors could be targeted in the MPER ana lyses to reduce these uncertainties. As the MPERs address these factors, better infonnation can be incorporated into the CE analyses and reflect the best available infonnation along with current uncertainties. It would help determine the confidence in projected aMW accomplishments. 54 There is the additional concern of knowledge being concentrated among a few key staff, if for some reason those staff members were not available to the Alliance in the future. Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report Recommendation #3 Treatment of Project Baselines (Dynamic versus Static): R3.- The baseline trend line is detennined in the planning process and not typically updated even when there have been substantial changes in the market (e., changes in the price of electricity) that would seem to have a clear impact on the number of energy efficient products that would be sold if the Alliance had not intervened in that market. R3.2 -Outside factors such as utility programs, prices, energy shortages do influence what would have happened without the Alliance program and it seems unreasonable to maintain that the baseline that was developed during program planning would remain the same going forward, even under changed circumstances. As a result, baselines should be dynamic in that they should be reconsidered every year and updated to reflect major market changes (e., higher electricity prices and constrained supplies that might lead customers to act) and the Alliances best judgment regarding how the baselines should be adapted over time. R3.3 -The baseline is one of the most influential and uncertain factors in producing any estimate of Alliance project effects. Best efforts are needed on baseline detennination, despite the complexities and uncertainties, and changes need to be made on 'an ongoing basis if Alliance claims are to be viewed as credible. Recommendation #Trade Ally Relationships Going Forward: R4.1 -Trade ally relationships are central to the Alliance objectives. Implementation contractors represent the Alliance to these important stakeholders and some contractors have proven to be a key factor in program success. Additional independent quality control processes should be implemented to ensure that good relations are consistently maintained. This can be as simple as having an Alliance staff member call an industry trade ally once a year just to make sure that contacts and coordination have been appropriate. Other potential activities might include a survey of retail partners, manufacturers, and utility key account reps regarding coordination of efforts, or development of guidelines for interactions with specific customer types or trade groups. While this relationship is covered in the process evaluation aspect of the MPERs work, these relationships are so central to Alliance objectives that an additional level of quality control and communication with trade ally organizations seems appropriate (Note: This was suggested by several parties as part of the stakeholder interview process). Recommendation #5 Guidance from the Board: R5.1 - Guidance is needed on what "claimed aMW impacts" means in the context of the MAR: Should the MAR only focus on market share indicators? , are these estimates meant to represent a 'best estimate" of aMW attributable to Alliance activities? Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report Should estimates be given a degree of confidence and/or expressed as range estimates to reflect the uncertainties in the attribution process. Should estimates in the MAR be supported by a specific ACE analysis to ensure consistency between savings estimates and levelized costs? These positions have implications on Alliance credibility and also for resources allocated to evaluation efforts. In addition, this recognizes that all business decisions am venture analysis both within and outside of the area of energy efficiency investments are made with uncertainty, and that precision in excess of what is used in the private sector to make good business decisions is not needed for the Alliance to successfully meet its objectives. 4 Final Comments This section is designed to again present the important context in which these issues are discussed and the recommendations made. The assessment perfonned by the study team found that the Alliance provides value trot exceeds its costs. Interviews with key stakeholders indicated that the Alliance: Operates well as a business organization Is strong in planning Communicates well Has impacted targeted markets. The benefit-cost analyses with the Team s revised numbers still show that the benefits of the Alliance have exceeded their costs. Overall, it is the study team s opinion that the reasons for establishing the Alliance are still valid and provide strong rationale for continuation: Energy markets invariably cut across utility and jurisdictional boundaries, it makes most sense to pursue these (MT) efforts regionally; and This regional approach by Alliance is an asset and can gain increased leverage by continuing its relationship building efforts with partners. Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report Table of Contents Appendices Individuals Interviewed , Interview Guides Pivot Assumption and Output Distributions for Scenario Analysis Document Log Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report Individuals Interviewed during Retrospective Evaluation Interview Guide - Organizational Effectiveness Interview Guide for Staff Interviews Task 2 Alternative Hypotheses Data Collection Instruments . CFL . Wi ndows . BOC . MagnaDrive Summit Blue Consulting Stratus Cons ulting NW Alliance Retrospective Final Report Task 2 Ad-hoc committee members Stan Price, NW Energy Efficiency Council Ken Canon Darlene Nimnich, Idaho Power John Savage, Oregon PUC Alliance staff Margie Gardner, executive director Dave Hewitt, Manager of CBI Jeff Harris, Manager, Development Andy Ekman, Project Coordinator Michael Ponder, Project Coordinator Susan Hermenet, Director, Planning and Implementation Marci Sanders, Manager, Residential Board members/others Norm Beckert, Consumer Representative, (retired - Boise Cascade) Tom Eckman, Conservation Manager, NWPPC Task 4 Staff and Committee Members Andy Eckman, Building Operator Certification program Jeff Harris, Cost Effectiveness models Ken Anderson, Cost Effectiveness models Marci Sanders, Energy Star Residential Lighting program Michael Ponder, Magnadrive program John Jennings, Energy Star Windows program Phil Degan, Energy Star Windows pro gram Ken Keating, BP A Prof!ram implementers My Ton, Sr. Research i\nalyst, Ecos Consulting Brian Simmons, Program Manager, Ecos Consulting Gary Curtis, West Wall Group Stan Price, NW Energy Efficiency Council Cynthia Putnam, NW Energy Efficiency Council Jim Cich, President, MagnaDrive Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report Prof!ram evaluators Jane Peters, President, Research Into Action, BOC program Marjorie McCrae, Research Into Action, BOC program Ken Seiden, Vice President, Quantec, MagnaDrive program Scott Dimontrosky, Quantec, Energy Star Windows program Stephen Grover, Sr. Economist, EcoNorthwest, Energy Star Residential Lighting Utility TJrOf!ram manaf!ers Jeff Bumgarner, Director - Energy Services, Pacificorp (Board Member) Bob Stolarski, Puget Sound Energy (Board Member) Other Stakeholders Market Actors Bill Nesmith, Oregon Energy Trust (Board Member) Tom Eckman, Conservation Manager, NWPPC Nathan Carpenter, Corporate Energy Manager, Boise Cascade Collin Cremo, Retailer, Costco Shana Cockerman, Retailer, Lowe Marc Orenschall, Reporter, News Data Corp. Nationalorf!anizations Rebecca Foster, Commercial Program Manager, Consortium for Energy Efficiency Richard Karney, Department of Energy Marsha Penhaker, Department of Energy Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report NEEA Retrospective Evaluation Interviews General Closinf! Questions Date This next section covers your views of the NW Alliance as a comprehensive business enterprise. We understand that our work effort is focused on examining the impacts of the Alliance on various markets and considerable work is being performed on this issue. In these questions, we want to quicky obtain general opinions about the effectiveness of the Alliance. We believe Specific reference is given to the issues being covered in this review, i.e., market effects of the Alliance: -------- focus the following on specifics/ interviewee can skip questions if N/ A-------------- Q 1: In your judgement, would the same effort have been placed on Market Transformation if the NW Alliance had not been formed? ( i.e - would the markets for energy efficiency products have been transformed anyway?) 1------------------------------------YES Q2: What is your opinion on how accurately the Alliance tracks overall aMW s for all of its program activities. For example, would you be surprised to find out that the true value of Alliance attained aMWs was (than reported in Market Activities Reportl. 50%less 25%less O%less 10%more 25%more 50%more Q3. How well has the Alliance chosen the ventures they have decided to participate in? 1------------------------------------POOR EXCELLENT Q4. How have they done with implementation/execution once a venture is chosen? 1. consumer programs? 1------------------------------------POOR EXCELLENT 2. entrepreneurial? 1-----------------------------------POOR EXCELLENT Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report 3. upstream? 1------------------------------------ 10POOR EXCELLENT 4. training? 1------------------------------------ 1 0POOR EXCELLENT Q5. Has the focus of the Alliance evaluation efforts been appropriate? 1------------- ----------------------- YES Q6. Overall, how well have they done on evaluation & attribution? 1------- ----------------------------- POOR EXCELLENT Q7. How well have they communicated to stakeholders on above program accomplishments? 1---- --------------------------------POOR EXCELLENT Q8. Have the right progress indicators been selected? 1------------------------------------ YES Q9. How does the Alliance do as a business organization? (in terms of the business activities they are engaged in.. 1------------------------------------POOR EXCELLENT Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report NEEA Retrospective Evaluation StaffInterviews General Questions These questions serve as a discussion guide for task 2 interviews with Alliance staff. cases, the questions were directed at specific programs 1. Role of project planning assumptions in development of strategy and theory b. Are Staff recommendation memos done b4 a program is launched? c. What is the role of the Cost Effectiveness Committee? d. What is the role of the Portfolio Committee? In some e. Do new projects go to the board for approval, or by committee? Are there sectors where MT may not be the best tool? g. What other tools have been considered by the Alliance to achieve the objectives? (The Alliance is by charter only interested in MT. Ifa utility wants a non-MT program, they don t run the money through the Alliance. 0 A specific emphasis will be placed on understanding baselines Assumptions regarding field performance of measures will also be reviewed h. Does the BOD receive hard numbers on cost effectiveness? ll.Selection of market progress indicators a. When in the process are the progress indicators selected? b. What impact targets are used? Is there a reason ALL programs with tracked savings don t have defined targets? c. How are NWPPC measure savings estimates used in planning and estimating program impacts? Is feedback given to them? d. CE Analysis -is payback, or NPV considered? How are the priorities for indicators picked? e. When a progress indicator is checked in the MAR, how has the staff determined that the indicator has been met? Role of the Market Assessment studies in providing a feedback loop for implementation llI. a. When is a market considered to be transformed? Is there a specific market penetration percentage, or change - that would indicate the work has been successful, and sustainable? b. How does the Alliance coordinate programs with national partners? CEE, Energy Star, etc. c. In the MAR, when it is indicated that the Alliance ranks regions - how are those regions defined? compared to other Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report VI. IV.Adaptive Management - how were programs modified to reflect evaluation results? a. How did the Alliance assess the value of other programs (utilities, BP A)? b. What do you ask Evaluation contractors to accomplish? Deliverables, scope? c. How is information gathered regarding the utility programs in the area? 2002 MAR indicates that 5MW was saved outside the Alliance for the entire region how was this number arrived at? d. How is in- field performance of savings verified? Time of use, persistence actually installed? e. How are the estimates of future savings (used on CE levelized cost numbers) calculated - what changes in baselines are assumed and how are they picked? How choices were made to retire certain programs. or morph them into new ones a. Are CE analyses done on a regular schedule? b. Does the portfolio committee make these decisions? Beginning the thought process that will look at alternative hypotheses regarding attribution of program results a. What has the Alliance done to try to 'bound' the savings estimates? b. What other factors do you think may have contributed to market changes for program X? Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report Al t e r n a t i v e B a s e l i n e H y p o t h e s i s (B O C ) En e r g y Un i t S a v i n g s Ve n t u r e I n f l u e n c e No n - v e n t u r e I n f l u e n c e Sa v i n g s #o f Ef f i c i e n c y Ve n t u r e E s t . Un i t s w/o Ve n t u r e Ve n t u r e Ve n t u r e Pe r i o d Lo w Me d Hi g h Lo w Me d Hi g h Lo w Me d Hi g h Ve n t u r e e s t . Lo w Me d Hi g h (k W h / S F / y ) Ve n t u r e Lo w Me d . Hi g h Es t . Es t . 20 0 2 5 ( . , s e e 10 0 % 8M W MP E R # 7 ) 15 . 3 M W ( c u m ) 16 . 8M W ( 2 0 1 0 ) Ke y v a r i a b l e s - t e s t H/ M / L r a n g e : Sq u a r e f e e t m a n a g e d , a v g = 2 3 4 00 0 Ba s e l i n e e n e r g y = 2 0 k w h / S f / y r Pe r s i s t e n c e = 5 y r s Se e ' as s u m p t i o n s Su m m i t B l u e C o n s u l t i n g St r a t u s C o n s u l t i n g NW A l l i a n c e R e t r o s p e c t i v e F i n a l R e p o r t Al t e r n a t i v e B a s e l i n e H y p o t h e s i s (C F L ) Un i t S a v i n g s Ad o p t i o n R a t e s Ve n t u r e I n f l u e n c e No n - v e n t u r e I n f l u e n c e Sa v i n g s En e r g y Ef f i c i e n c y Ve n t u r e wl o Ve n t u r e Ve n t u r e Ve n t u r e Ve n t u r e Ac t i o n P e r i o d Es t . Lo w Me d Hi g h #o f Ve n t u r e Es t . Lo w Me d Hi g h Es t . Lo w Me d Hi g h Lo w Me d Hi g h (k W h / y ) Un i t s Lo w Me d . Hi g h es t . 20 0 1 6M - 10 0 % 31 . 1 co u p o n - g I v e aw a y = Ke y v a r i a b l e s - t e s t H/ M / L r a n g e : In s t a l l a t i o n r a t e = 1 0 0 % Di s p l a c e d w a t t s = 7 4 Ho u r s p e r d a y = 3 i n t e r i o r Sp i l l o v e r = 3 6 % ( f r o m A l l i a n c e t o c o u p o n p r o g r a m ) Se e ' as s u m p t i o n s 5 e x t e r i o r ( s e e : n a t i o n a l s t u d y ) Su m m i t B l u e C o n s u l t i n g St r a t u s C o n s u l t i n g NW A l l i a n c e R e t r o s p e c t i v e F i n a l R e p o r t Al t e r n a t i v e B a s e l i n e H y p o t h e s i s (M a g n a D r i v e ) En e r g y Un i t S a v i n g s Ve n t u r e I n f l u e n c e No n - v e n t u r e I n f l u e n c e Sa v i n g s Ef f i c i e n c y Ve n t u r e # o f wlo Ve n t u r e Ac t i o n Lo w Me d Hi g h Un i t s Ve n t u r e Lo w Me d Hi g h . Ve n t u r e Lo w Me d Hi g h Ve n t u r e Lo w Me d Hi g h Es t . Pe r i o d Ve n t u r e Lo w Me d . Hi g h Es t . Es t . es t . (k W h l H P ) 20 0 1 10 0 % 1. 4 3 85 3 aM W Ke y v a r i a b l e s - t e s t H/ M / L r a n g e : Ho r s e p o w e r i n s t a l l e d = 1 4 71 6 ( t h r o u g h 2 0 0 1 ) VS D m a r k e t g r o w t h = 5 % Sa v i n g s p e r H P = 8 5 3 k W h Pe r s i s t e n c e Se e ' a s s u m p t i o n s Su m m i t B l u e C o n s u l t i n g St r a t u s C o n s u l t i n g NW A l l i a n c e R e t r o s p e c t i v e F i n a l R e p o r t Al t e r n a t i v e B a s e l i n e H y p o t h e s i s ( W i n d o w s ) En e r g y Un i t S a v i n g s Ma r k e t P e n e t r a t i o n Ve n t u r e I n f l u e n c e No n - v e n t u r e I n f l u e n c e Sa v i n g s # o f Ef f i c i e n c y A c t i o n Ve n t u r e E s t . Un i t s wlo Ve n t u r e Ve n t u r e Ve n t u r e Ve n t u r e Pe r i o d Lo w Me d Hi g h Lo w Me d Hi g h Lo w Me d Hi g h Lo w Me d Hi g h (k W h / S F / y ) Ve n t u r e Lo w Me d . Hi g h Es t . Es t . es t . 20 0 2 66 % 14 % 10 0 % 13 . 8M W (a v g . ba s e d or 7 0 % (c u m ' 02 ) on m o d e l ? ) (2 0 0 2 ) 40 % 45 . 7M W or 8 0 % (2 0 1 0 ) (2 0 1 0 ) Ke y v a r i a b l e s - t e s t H/ M / L r a n g e : Ho u s i n g S t o c k ( e l e c h e a t e d , s i n g l e f a m i l y = 4 0 % Mu l t i - f a m = 82 % Ma n u f a c t u r e d = 9 4 % In c r e m e n t a l C o s t of w i n d o w s = $ . 4 8 / S F Li f e o f W i n d o w = 6 0 y e a r s Se e ' as s u m p t i o n s Su m m i t B l u e C o n s u l t i n g St r a t u s C o n s u l t i n g NW A l l i a n c e R e t r o s p e c t i v e F i n a l R e p o r t PIVOT ASSUMPTION AND OUTPUT DISTRIBUTIONS FOR SCENARIO ANAL YS IS Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report Energy Star Lighting Input Distributions Distribution for Displaced watts/N 138 060. 050 040 030 020 010 000 10% 47.4611 70.6919 120 10'% Distribution for Hours On/Day / Interior/B 135 900. 800 700 600 500 0.400 300 0.200 100 000 125 2.75 3.375 10% 10%1328 3.3463 Distribution for Hours On/Day / Exterior/C135 350. 300 250 200 150 100 050 000 10%2.4167 5.5143 10% Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report Distribution for Removal nterior/B 141 060. Mean=27.98696 050 040 030 020 010 000 10% 16.4537 39.9488 10% Distribution for Lifetime (Hours) Iinterior/B136 500 000 c:: 2.500 c: 2.000 ~ 1.500 ::J~ 1.000 500 000 800-. 10. Values in Thousands 10% 9999 9.026 10% 2003 Lamp Cost 45 10'/0 10'/04~2 5.6E3 Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report 800 700 600 500 400 300 200 100 000 Distribution for 2007 / per Lamp/D22 10% 10%3915 3.6083 High Influence Assumptions 000. 500 ~ 3.000 c::~ 2.500 .- 2.000 ::J 1.500 ~ 1.000 500 000 200 LO 1.000 ~ 0.800 ..- .- 0.600 0.400 0.200 000 Distribution for Baseline 2001/C45 10% Values in Millions 10%7321 5.6517 Distribution for Baseline 2002/C46 100 140 180 Values in Thousands10% 10% 58.3318 141.6441 Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report 000. 500 ";" 3.000 c::~ 2.500 .- 2.000 :J 1.500 ~ 1.000 500 000 Distribution for Baseline 2001/C45 10% Values in Millions 10%7321 5.6517 Low Influence Assumptions I'-- 5 .- 3 c: 5 en Distribution for Baseline 2001/C45 375 125 Values in Millions 10%1845 3.0455 10% Distribution for Baseline 2002/C46 112.175 237.300 Values in Thousands10% 10% 119.124 254.4268 Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report 8-4 High Influence Results 030-. 0.01 0.01 2010 Cumulative Savings tv1ea1=741W21 000 ED 100 1(13/0 ~4078 00-7S3 700-. 000 '\i()100 1(13/0 Venture + Post TRC Levelized Cost 1(13/0 ID19 21167 1(13/0 Distribution for CE Venture period, TRC/F87 600 1.400 200 000 800 600 0.400 200 000 2.4 10% 8395 1.5093 10% Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report 1.400-. 000 Ventrue + Post TR CE Index 1.125 1.7510028 1mD 2375 Venture Period TR CE INDEX 1.47Ol2 1.6162 1.9 Venture + Post Alliance CE 10 5% 7ffJT1 175192 8'/0 Summit Blue Consulting Stratus Consulting 8'/0 NW Alliance Retrospective Final Report Low Influence Results 2010 Cumulative Savings Distribution for Cumulative aMW savings/FaD 030 025 020 015 010 005 000 125 160 10%43.4121 80.2613 10% Venture + Post TR levelized cost 600-. 9179 3.4779 5'/0 Venture only TR Levelized cost 000 10% 2.0793 5.1 1(1/0 Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report 500-. Venture only Alliance levelized cost 0.3 0. 10'/0 1467 ID18 10'/0 Venture + Post TR CE 1.4 10'/0 ffi:18 1 AOO3 1.9 2.4 10'/0 Distribution for CE Venture period , TRC/F87 500. Mean=O.87093 000 500 000 500 000 1.4 10%6473 1.1189 10% Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report Windows Input Distributions .,. Elec. Space Heat Saturation .-- r- I --- 0.4 .... 3471 .4941 1OJ/o 1OJ/o Distribution for Weighted Cost per SF window / Baseline/... .:IW s:m 10"10 10"/0 Output Distributions 350-, 000 Cumulative Savings 2002 a1:I12 115714 10"/0 10"/0 Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report Distribution for 2010 Cumulative aMW Savings/D86 250 0.200 150 100 050 000 001-0. 0QO; 000 Mean=~.07602 .-- 90% --...----....- 27.3568 32.8051 Venture + Post Levelized Cost TR 1v'mFO.6'.mm 025 1OJ/o1OJ/o 3864 . Venture onl TR Levelized Cost 1.25 10%606 .9441 10% Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report Distribution for Venture + Post TR CE/F88 o.c)OO 10% 1184 3.3313 1(1110 Distribution for Venture only TR CElF89 800- 600 1.400 200 000 800 600 0.400 200 000 ----"; j I i 'r 1.875 625 1(11/0 0387 2.6661 1(11/0 Venture + Post Alliance Levelized Cost -0.445 1CWo II!II ~ - 4502 -0.-0.375 -0. 1(11/0 Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report Venture only Alliance Levelized Cost -0.325 10'/0 361 -3056 -0.025 10'/0 Distribution for Venture + Post Alliance CE/F91 10'/0 120 160 68.7474 98.0016 10'/0 10'/0 Summit Blue Consulting Stratus Consulting Distribution for Venture only Alliance CE/F92 1/Di nrm -.--- 15.6315 222516 10'10 NW Alliance Retrospective Final Report BOC Input Distributions Square Footage 6-. c:: ..- .2 2 ;::::. 1 00 14) Values in Thousands 100 r------ - -----------~ --- .... OO'/o ..--~---_.._-__---------- . 11Iril8 ffi.4€ffi 216.'3475 Distribution for Savings per SFIB25 M:aF1 f3333 0.5 1.5 -- -- -- 00% - - -- -. 4$1 1.87'22 Distribution for Life (years )/B27 250-. 1CWo 3.5:18 10'/0 Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report Output Distributions Distribution for 2002 aMW savingslF74 0.01 000 15 :D 10'/0 10'/09.5871 :nwJ Distribution for 2010 Venture Cum. aMW SavingsIF82 ffi67 EiHlI- -p- --- ~ W/o - - -- 7 f1525 46.00 Venture + Post TR Levelized Cost 1.200-. 000 -0.0.5 10% z:m 4238 10'10 ;.. Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report Distribution for Venture Period Only Levelized cost TRClE89 000 .Q25 0.5 125 1(11/0 re34 1D13 1(11/0 Venture + Post TR CE Index 120-. 10 1(11/0 3005 142514 1(11/0 Distribution for Venture period only TRC CElF89 0.20 000 1(11/0 1.7764 82m 1(11/0 Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report Distribution for Venture + Post Levelized Cost Alliance/E91 500... 000 .0.0.5 10'/0 D'83m -1192 10'/0 ;:. Distribution for Venture Period Only Levelized Cost All... 1200-.. 000 IE8J 2133 IIJJl 10'/0 10'/0 ;:. Distribution for Venture + Post Alliance CE/F91 040... 000 10'/0 9.3794 43.6111 10'10 Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report Venture only Alliance CE 120-. MiBF8Wffi7 :;D 1(11/0 3.1312 14fHJ2 1(11/0 MagnaDrive Input Assumptions Distribution for Market growth/B 135 X COO.Xc=4. 95"10 Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report Output Results 250-, 000 Distribution for 2010 Cumulative aMW Savings/D83 10% 21.3849 27.1402 10% Venture + Post TR Levelized Cost 125 13125 1375 10% 1.3E 1.37a5 1 A375 10% Venture + Post Alliance Levelized Costs 30-. ro- 1- c- .0..0.11 .000 1001 -f.J1Z3 .o. 10'/0 10'/0 Summit Blue Consulting Stratus Consulting .om NW Alliance Retrospective Final Report 14, 1.7 Disbibution for Venture + Post TR CE/F88 1.775 185 1.925 10Vo 1.84ffi 1.9276 1(11/0 Distribution for Venture + Post Alliance CE/F91 1(11/0 192132 25ffi19 (11/0 Distribution for Venture Only Levelized Cost TRP/E92 c::=JBa 4. 766~ 11.8765 10% Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report DOCUMENT LOG Summit Blue Consulting Stratus Consulting NW Alliance Retrospective Final Report Ca t e g o r y Su b c a t e g o r y Do c u m e n t N a m e Da t e R e c e i v e d Fr o m W h e r e Au t h o r ( s ) Da t e o f R e p o r t An n u a l R e p o r t 19 9 9 A n n u a l R e p o r t Ef f i c i e n c y O p p o r t u n i t i e s i n S i l i c o n C r y s t a l 5/ 8 / 2 0 0 3 N E E A w e b s i t e Si e m e n s S o l a r I n d u s t r i e s No d a t e Gr o w i n g F a c i l i t i e s Gr e g M i h a l i k Re d e s i g n o f C r y s t a l G r o w e r H o t Z o n e s Br y a n F i c k e t t As s e s s m e n t No v - As s e s s m e n t As s e s s m e n t Da t a R e q u e s t Da t a R e q u e s t Da t a R e q u e s t Da t a R e q u e s t Da t a R e q u e s t Ch a p t e r 6 Dr a f t F o u r t h N o r t h w e s t Po w e r P l a n Co m p r e h e n s i v e R e v i e w o f F i n a l R e p o r t th e N o r t h w e s t E n e r g y To w a r d a C o m p e t i t i v e E l e c t r i c P o w e r Sy s t e m In d u s t r y f o r t h e 2 1 s t C e n t u r y De c e m b e r 1 2 , 1 9 9 6 Co m p r e h e n s i v e C o n t a c t R e p o r t Am o n g R e g i o n a l P a r t i e s i n S u p p o r t o f t h e No r t h w e s t E n e r g y E f f i c i e n c y A l l i a n c e Oc t o b e r 1 5 , 1 9 9 9 Ba s e l i n e A s s e s s m e n t Ex e c u t i v e S u m m a r y Ba s e l i n e M a r k e t As s e s s m e n t E x e c u t i v e Su m m a r y Ma r k e t A s s e s s m e n t Al l i a n c e P o r t f o l i o Co n t a c t R e p o r t Me m o r a n d u m o f Ag r e e m e n t En e r g y S t a r R e s i d e n t i a l L i g h t i n g F i x t u r e s Pr o g r a m M a r k e t P r o g r e s s E v a l u a t i o n Re p o r t # 1 Su p e r G o o d C e n t s M a n u f a c t u r e d H o u s i n g Ve n t u r e Ex e c u t i v e S u m m a r y 5/8 / 2 0 0 3 N E E A w e b s i t e 5/ 8 / 2 0 0 3 N E E A w e b s i t e Pa c i f i c C o n s u l t i n g S e r v i c e s Sh e l F e l d m a n M a n a g e m e n t C o n s u l t a n t s Pa c i f i c E n e r g y A s s o c i a t e s , I n c . Da v e H e w i t t Je f f P r a t t Ga r y S m i t h Ea s t o n C o n s u l t a n t s , I n c . Xe n e r g y , ln c . Au g - De c - Op p o r t u n i t i e s f o r I n d u s t r i a l M o t o r S y s t e m s in t h e P a c i f i c N o r t h w e s t All i a n c e P o r t f o l i o C o s t E f f e c t i v e n e s s ( 2 0 0 2 Nu m b e r s ) Re s o u r c e I s s u e s i n C o m p e t i t i v e M a r k e t s 5/ 8 / 2 0 0 3 N E E A w e b s i t e 4/2 9 / 2 0 0 3 M a r g i e G a r d n e r 5/1 2 / 2 0 0 3 L i z S a u n d e r s De c - Ma y - o 3 Oc t - Da t a R e q u e s t De c - Da t a R e q u e s t Da t a R e q u e s t Me m o r a n d u m o f Ag r e e m e n t Op e r a t i o n a l A u d i t Ov e r v i e w 5/ 1 2 / 2 0 0 3 L i z S a u n d e r s 5/ 1 2 / 2 0 0 3 L i z S a u n d e r s 4/ 2 9 / 2 0 0 3 M a r g i e G a r d n e r No r t h w e s t E n e r g y E f f i c i e n c y A l l i a n c e Oc t o b e r 3 1 , 19 9 6 No r t h w e s t E n e r g y E f f i c i e n c y A l l i a n c e Op e r a t i o n a l A u d i t De c e m b e r 2 1 , 1 9 9 8 Ov e r v i e w o f C o n c l u d e d A l l i a n c e P r o j e c t s 4/ 2 9 / 2 0 0 3 M a r g i e G a r d n e r Oc t - Pr i c e W a t e r h o u s e C o o p e r s No r t h w e s t E n e r g y E f f i c i e n c y A l l i a n c e No r t h w e s t E n e r g y E f f i c i e n c y A l l i a n c e Da t a R e q u e s t Ov e r v i e w 5/ 1 2 / 2 0 0 3 L i z S a u n d e r s 4/2 9 / 2 0 0 3 M a r g i e G a r d n e r Ov e r v i e w o f C u r r e n t A l l i a n c e P r o j e c t s 4/ 2 9 / 2 0 0 3 M a r g i e G a r d n e r Da t a R e q u e s t Ev a l u a t i o n Ev a l u a t i o n Ev a l u a t i o n Ev a l u a t i o n St a f f R e c o m m e n d a t i o n s Ba s e l i n e M a r k e t Ev a l u a t i o n Ma r k e t B a s e l i n e Ev a l u a t i o n Ma r k e t B a s e l i n e Ev a l u a t i o n Ma r k e t B a s e l i n e Ev a l u a t i o n St a f f R e c o m m e n d a t i o n M e m o r a n d a (M u l t i p l e ) Ev a p o r a t o r F a n V F D I n i t i a t i v e 4/ 2 9 / 2 0 0 3 M a r g i e G a r d n e r 5/ 8 / 2 0 0 3 N E E A w e b s i t e Ma c r o I n t e r n a t i o n a l , I n c . Ap r - Ba c G e n B i o W i s e P r o j e c t , N o . 1 ( 3 / 0 0 ) 5/ 8 / 2 0 0 3 N E E A w e b s i t e Qu a n t u m C o n s u l t i n g , I n c . Ad o l f s o n A s s o c i a t e s , I n c . Re s e a r c h I n t o A c t i o n , I n c . Ja n e S . P e t e r s , P h D Ma ~ o r i e R . M c R a e , P h D De t h m a n & T a n g o r a , L L C Lin d a D e t h m a n Pa c i f i c C o n s u l t i n g S e r v i c e s Ma r - Ma r - Ma r - Ef f i c i e n t B u i l d i n g P r a c t i c e s I n i t i a t i v e , N o . 5/ 8 / 2 0 0 3 N E E A w e b s i t e Su m m i t B l u e C o n s u l t i n g St r a t u s C o n s u l t i n g EZ S i m : B i l l i n g S i m u l a t i o n f o r S m a l l Co m m e r c i a l F a c i l i t i e s 5/8 / 2 0 0 3 N E E A w e b s i t e NW A l l i a n c e R e t r o s p e c t i v e F i n a l R e p o r t Ev a l u a t i o n Ma r k e t B a s e l i n e Fa n S p e e d R e d u c t i o n i n P n e u m a t i c 5/ 8 / 2 0 0 3 N E E A w e b s i t e SB W C o n s u l t i n g , I n c . De c - Ev a l u a t i o n Co n v e y i n g S y s t e m s i n t h e S e c o n d a r y Qu a n t u m C o n s u l t i n g , I n c . Wo o d P r o d u c t s I n d u s t r y , N o . 1 ( 1 2 / 9 9 ) Ar g o B l o w e r & M f g . C o . Ev a l u a t i o n Ma r k e t B a s e l i n e No r t h w e s t R e s i d e n t i a l D u c t s 5/ 8 / 2 0 0 3 N E E A w e b s i t e XE N E R G Y , I n c . Ja n - Ev a l u a t i o n Ev a l u a t i o n Ma r k e t B a s e l i n e Pe r f o r m a n c e - Te s t e d C o m f o r t S y s t e m s , N o . 5/ 8 / 2 0 0 3 N E E A w e b s i t e XE N E R G Y , I n c . De c - Ev a l u a t i o n Ev a l u a t i o n Ma r k e t B a s e l i n e En e r g y S t a r R e s i d e n t i a l L i g h t i n g F i x t u r e s 5/8 / 2 0 0 3 N E E A w e b s i t e Pa c i f i c C o n s u l t i n g S e r v i c e s Au g - Ev a l u a t i o n E x e c . Su m m a r y Ev a l u a t i o n Ma r k e t P r o g r e s s Ar c h i t e c t u r e & E n e r g y P r o g r a m , F i n a l 5/ 8 / 2 0 0 3 N E E A w e b s i t e Re s e a r c h I n t o A c t i o n , I n c . Ju n - Ev a l u a t i o n Re p o r t Ja n e S . P e t e r s , P h D Ma r j o r i e R . M c R a e , P h D Ev a l u a t i o n Ma r k e t P r o g r e s s Ar c h i t e c t u r e & E n e r g y P r o g r a m , N o . 5/ 8 / 2 0 0 3 N E E A w e b s i t e Re s e a r c h I n t o A c t i o n , I n c . Au g - Ev a l u a t i o n Ja n e S . P e t e r s , P h D Ma r j o r i e R . M c R a e , P h D Ev a l u a t i o n Ma r k e t P r o g r e s s Bu i l d i n g O p e r a t o r C e r t i f i c a t i o n , N o . 3 ( 5 / 0 0 ) 5/8 / 2 0 0 3 N E E A w e b s i t e Re s e a r c h I n t o A c t i o n , I n c . Ma y - Ev a l u a t i o n Ja n e S . P e t e r s . P h D Sh a r o n A . B a g g e t t , P h D Ma r j o r i e R . M c R a e St e l l a r P r o c e s s e s , I n c . Da v e R o b i s o n , P E Ev a l u a t i o n Ma r k e t P r o g r e s s Bu i l d i n g O p e r a t o r C e r t i f i c a t i o n , N o . 5/ 8 / 2 0 0 3 N E E A w e b s i t e Re s e a r c h I n t o A c t i o n , I n c . Ma r - Ev a l u a t i o n Ja n e S . P e t e r s , P h D Ma r j o r i e R . M c R a e , P h D G. F l y n n Ev a l u a t i o n Ma r k e t P r o g r e s s Bu i l d i n g O p e r a t o r s C e r t i f i c a t i o n P r o g r a m . 5/ 8 / 2 0 0 3 N E E A w e b s i t e Re s e a r c h I n t o A c t i o n , I n c . Ma y - Ev a l u a t i o n Wa s h i n g t o n S t a t e , R e p o r t S u m m a r y Ja n e S . P e t e r s . P h D Sh a r o n A . B a g g e t t , P h D St e l l a r P r o c e s s e s , I n c . Da v e R o b i s o n , P E Ev a l u a t i o n Ma r k e t P r o g r e s s Co m m i s s i o n i n g I n P u b l i c B u i l d i n g s P r o j e c t , 5/ 8 / 2 0 0 3 N E E A w e b s i t e Qu a n t u m C o n s u l t i n g , I n c . De c - Ev a l u a t i o n No . 2 ( 1 2 / 9 9 ) Ev a l u a t i o n Ma r k e t P r o g r e s s Co m m i s s i o n i n g I n P u b l i c B u i l d i n g s P r o j e c t , 5/ 8 / 2 0 0 3 N E E A w e b s i t e Qu a n t u m C o n s u l t i n g , I n c . Fe b - Ev a l u a t i o n No . Ev a l u a t i o n Ma r k e t P r o g r e s s Dr i v e P o w e r I n i t i a t i v e , N o . 1 ( 3 / 0 0 ) 5/ 8 / 2 0 0 3 N E E A w e b s i t e Pa c i f i c E n e r g y A s s o c i a t e s , I n c . Ma r - Ev a l u a t i o n Je n n i f e r S t o u t Fr e d G o r d o n St e v e n S c o t t , P E Ev a l u a t i o n Ma r k e t P r o g r e s s Dr i v e P o w e r I n i t i a t i v e , N o . 5/ 8 / 2 0 0 3 N E E A w e b s i t e Pa c i f i c E n e r g y A s s o c i a t e s , I n c . Fe b - Ev a l u a t i o n Je n n i f e r S t o u t Fr e d G o r d o n St e v e n S c o t t , P E Ev a l u a t i o n Ma r k e t P r o g r e s s Dr i v e P o w e r I n i t i a t i v e , N o . 5/ 8 / 2 0 0 3 N E E A w e b s i t e Pa c i f i c E n e r g y A s s o c i a t e s , I n c . No v - Ev a l u a t i o n Je n n i f e r S t o u t St e v e n S c o t t Fr e d G o r d o n Ev a l u a t i o n Ma r k e t P r o g r e s s Dr i v e P o w e r I n i t i a t i v e , N o . 5/ 8 / 2 0 0 3 N E E A w e b s i t e Cu r r e n t s C o n s u l t i n g Ma r - Ev a l u a t i o n Je n n i f e r S t o u t St e v e n S c o t t Su m m i t B l u e C o n s u l t i n g NW A l l i a n c e R e t r o s p e c t i v e F i n a l R e p o r t St r a t u s C o n s u l t i n g Ev a l u a t i o n Ma r k e t P r o g r e s s Ef f i c i e n t B u i l d i n g P r a c t i c e s I n i t i a t i v e , N o . 5/ 8 / 2 0 0 3 N E E A w e b s i t e De t h m a n & T a n g o r a , L L C Ju l - O 1 Ev a l u a t i o n Fi n a l R e p o r t Lin d a D e t h m a n Re s e a r c h I n t o A c t i o n , I n c . Ja n e S . P e t e r s , P h D Ma r j o r i e R . M c R a e , P h D Ev a l u a t i o n Ma r k e t P r o g r e s s En e r g y I d e a s C l e a r i n g h o u s e 5/ 8 / 2 0 0 3 N E E A w e b s i t e Qu a n t e c , L L C Ju n - Ev a l u a t i o n Sc o t t D i m e t r o s k y Mi c h a e l L u e v a n e M. S a m i K h a w a j a , P h D Ev a l u a t i o n Ma r k e t P r o g r e s s En e r g y I d e a s C l e a r i n g h o u s e 5/ 8 / 2 0 0 3 N E E A w e b s i t e Qu a n t e c Ma y - O O Ev a l u a t i o n Sc o t t D i m e t r o s k y M. S a m i K h a w ~ a , P h D Ju p i t e r C o m m u n i c a t i o n s Gi l m o r e R e s e a r c h G r o u p Ev a l u a t i o n Ma r k e t P r o g r e s s En e r g y I d e a s C l e a r i n g h o u s e , N o . 1 ( 1 2 / 9 9 ) 5/8 / 2 0 0 3 N E E A w e b s i t e Qu a n t e c De c - Ev a l u a t i o n Ju p i t e r C o m m u n i c a t i o n s Gi l m o r e R e s e a r c h G r o u p Ev a l u a t i o n Ma r k e t P r o g r e s s En e r g y I d e a s C l e a r i n g h o u s e , N o . 5/ 8 / 2 0 0 3 N E E A w e b s i t e Qu a n t e c , L L C Fe b - Ev a l u a t i o n Ev a l u a t i o n Ma r k e t P r o g r e s s En e r g y S t a r R e s i d e n t i a l L i g h t i n g F i x t u r e 5/ 8 / 2 0 0 3 N E E A w e b s i t e Pa c i f i c C o n s u l t i n g S e r v i c e s Au g - Ev a l u a t i o n Pr o g r a m , N o . 2 ( 8 / 9 9 ) Sh e l F e l d m a n M a n a g e m e n t C o n s u l t a n t s Ev a l u a t i o n Ma r k e t P r o g r e s s En e r g y S t a r R e s o u r c e - Ef f i c i e n t C l o t h e s 5/ 8 / 2 0 0 3 N E E A w e b s i t e Pa c i f i c E n e r g y A s s o c i a t e s , I n c . Ma y - Ev a l u a t i o n Wa s h e r s , N o . Da v e H e w i t t Je f f P r a t t Ga r y S m i t h Ev a l u a t i o n Ma r k e t P r o g r e s s En e r g y S t a r R e s o u r c e - Ef f i c i e n t C l o t h e s 5/8 / 2 0 0 3 N E E A w e b s i t e Pa c i f i c E n e r g y A s s o c i a t e s , I n c . No v - Ev a l u a t i o n Wa s h e r s , N o . Da v e H e w i t t Je f f P r a t t Ga r y S m i t h Ev a l u a t i o n Ma r k e t P r o g r e s s En e r g y S t a r W i n d o w s P r o g r a m , N o . 5/ 8 / 2 0 0 3 N E E A w e b s i t e Qu a n t e c Ma r - Ev a l u a t i o n (3 / 0 0 ) Ev a l u a t i o n Ma r k e t P r o g r e s s En e r g y S t a r W i n d o w s P r o g r a m , N o . 5/ 8 / 2 0 0 3 N E E A w e b s i t e Qu a n t e c No v - Ev a l u a t i o n Ev a l u a t i o n Ma r k e t P r o g r e s s En e r g y S t a r W i n d o w s , N o . 2 ( 8 / 9 9 ) 5/ 8 / 2 0 0 3 N E E A w e b s i t e Qu a n t e c Au g - Ev a l u a t i o n Ev a l u a t i o n Ma r k e t P r o g r e s s En e r g y S t a r W i n d o w s , N o . 5/ 8 / 2 0 0 3 N E E A w e b s i t e Qu a n t e c Ja n - Ev a l u a t i o n Ev a l u a t i o n Ma r k e t P r o g r e s s Ev a p o r a t o r F a n V F D I n i t i a t i v e , N o . 5/ 8 / 2 0 0 3 N E E A w e b s i t e Ma c r o I n t e r n a t i o n a l , I n c . No v - Ev a l u a t i o n Ev a l u a t i o n Ma r k e t P r o g r e s s Ev a p o r a t o r F a n V F D I n i t i a t i v e , N o . 5/ 8 / 2 0 0 3 N E E A w e b s i t e Pa c i f i c E n e r g y A s s o c i a t e s , I n c . Ju n - Ev a l u a t i o n St e v e n S c o t t , P E Fr e d G o r d o n Me t a R e s o u r c e G r o u p Ev a l u a t i o n Ma r k e t P r o g r e s s EZ S i m : B i l l i n g S i m u l a t i o n f o r S m a l l 5/8 / 2 0 0 3 N E E A w e b s i t e Pa c i f i c C o n s u l t i n g S e r v i c e s Oc t - Ev a l u a t i o n Co m m e r c i a l F a c i l i t i e s , N o . Ev a l u a t i o n Ma r k e t P r o g r e s s Li g h t i n g D e s i g n L a b 5/ 8 / 2 0 0 3 N E E A w e b s i t e En e r g y M a r k e t I n n o v a t i o n s , I n c . Se p - Ev a l u a t i o n Ev a l u a t i o n Ma r k e t P r o g r e s s Li g h t i n g D e s i g n L a b , N o . 5/ 8 / 2 0 0 3 N E E A w e b s i t e En e r g y M a r k e t I n n o v a t i o n s , I n c . Ap r - Ev a l u a t i o n He s c h o n g M a h o n e G r o u p Su m m i t B l u e C o n s u l t i n g NW A l l i a n c e R e t r o s p e c t i v e F i n a l R e p o r t St r a t u s C o n s u l t i n g Ev a l u a t i o n Se p - Ev a l u a t i o n Ev a l u a t i o n Ev a l u a t i o n Ev a l u a t i o n Ma r k e t P r o g r e s s Ev a l u a t i o n Ma r k e t P r o g r e s s Ev a l u a t i o n Ma r k e t P r o g r e s s Ev a l u a t i o n Ma r k e t P r o g r e s s Ev a l u a t i o n Ma r k e t P r o g r e s s Ev a l u a t i o n Li g h t W i s e , N o . 2 ( 9 / 9 9 ) 5/8 / 2 0 0 3 N E E A w e b s i t e De t h m a n & A s s o c i a t e s Li n d a D e t h m a n Gi l m o r e R e s e a r c h G r o u p Qu a n t e c Ma r k e t L i n k S t r a t e g i e s Sc h i l l e r A s s o c i a t e s XE N E R G Y Qu a n t e c , L L C Sc h i l l e r A s s o c i a t e s Qu a n t e c , L L C La u r e n M i l l e r M. S a m i K h a w ~ a , P h D Re s e a r c h I n t o A c t i o n , I n c . Ja n e S . P e t e r s , P h D Ma ~ o r i e R . M c R a e Sh e l F e l d m a n M a n a g e m e n t C o n s u l t a n t s Sh e l F e l d m a n , P h D Ma y - Ma y - Ap r - No v - Ev a l u a t i o n Ju n - Ma r k e t P r o g r e s s Ev a l u a t i o n Ma g n a D r i v e , N o . 5/ 8 / 2 0 0 3 N E E A w e b s i t e Sh e l F e l d m a n M a n a g e m e n t C o n s u l t i n g Sh e l F e l d m a n , P h D Ev a l u a t i o n No v - Ma r k e t P r o g r e s s Ev a l u a t i o n Ma g n a D r i v e , N o . 5/ 8 / 2 0 0 3 N E E A w e b s i t e Sh e l F e l d m a n M a n a g e m e n t C o n s u l t i n g Sh e l F e l d m a n Ev a l u a t i o n Ja n - Ev a l u a t i o n Ev a l u a t i o n Ev a l u a t i o n Ev a l u a t i o n Ev a l u a t i o n Ma r k e t P r o g r e s s Ev a l u a t i o n Ma r k e t P r o g r e s s Ev a l u a t i o n Ma r k e t P r o g r e s s Ev a l u a t i o n Ma r k e t P r o g r e s s Ev a l u a t i o n Ma r k e t P r o g r e s s Ev a l u a t i o n Ma r k e t P r o g r e s s Ev a l u a t i o n Ma r k e t P r o g r e s s E v a l u a t i o n R e p o r t , N o . Fin a l R e p o r t 5/ 8 / 2 0 0 3 N E E A w e b s i t e Pa c i f i c E n e r g y A s s o c i a t e s , I n c . Fr e d G o r d o n Jo h n J e n n i n g s Le s T u m i d a j Wi l l M i l l e r Re s e a r c h I n t o A c t i o n , I n c . Sh a r o n A . B a g g e t t , P h D Ja n e S . P e t e r s , P h D Re s e a r c h I n t o A c t i o n , I n c . Sh a r o n B a g g e t t , P h D Ja n e S . P e t e r s , P h D St e l l a r P r o c e s s e s , I n c . Da v e R o b i s o n , P E Re s e a r c h I n t o A c t i o n , I n c . Ja n e S . P e t e r s , P h D Sh a r o n A . B a g g e t t , P h D Re s e a r c h I n t o A c t i o n , I n c . Ja n e S . P e t e r s , P h D Ma r j o r i e R . M c R a e , P h D St e l l a r P r o c e s s e s , I n c . Da v e R o b i s o n , P E Pa c i f i c E n e r g y A s s o c i a t e s , I n c . St e v e n S c o t t , P E Je n n i f e r S t o u t Fr e d G o r d o n Oc t - Ma y - Ju l - Se p - Ju n - O O Su m m i t B l u e C o n s u l t i n g St r a t u s C o n s u l t i n g Mi c r o e l e c t r o n i c s I n i t i a t i v e , N o . 5/ 8 / 2 0 0 3 N E E A w e b s i t e No r t h w e s t E n e r g y E d u c a t i o n I n s t i t u t e , N o . 5/ 8 / 2 0 0 3 N E E A w e b s i t e No r t h w e s t E n e r g y E d u c a t i o n I n s t i t u t e , N o . 5/ 8 / 2 0 0 3 N E E A w e b s i t e Pr e m i u m E f f i c i e n c y M o t o r s ( E x e c . Su m m a r y ) 5/ 8 / 2 0 0 3 N E E A w e b s i t e Re g i o n a l B u i l d i n g O p e r a t o r C e r t i f i c a t i o n Ex e c . S u m m a r y 5/8 / 2 0 0 3 N E E A w e b s i t e Re g i o n a l B u i l d i n g O p e r a t o r C e r t i f i c a t i o n No . , V o l u m e 1 5/ 8 / 2 0 0 3 N E E A w e b s i t e Re g i o n a l B u i l d i n g O p e r a t o r C e r t i f i c a t i o n , No . 2, V o l u m e 2 5/ 8 / 2 0 0 3 N E E A w e b s i t e Re g i o n a l B u i l d i n g O p e r a t o r C e r t i f i c a t i o n No . 5/ 8 / 2 0 0 3 N E E A w e b s i t e Sa v - Air M a r k e t T r a n s f o r m a t i o n I n i t i a t i v e 5/ 8 / 2 0 0 3 N E E A w e b s i t e NW A l l i a n c e R e t r o s p e c t i v e F i n a l R e p o r t C- 4 Ev a l u a t i o n Fe b - Ev a l u a t i o n Ev a l u a t i o n Ev a l u a t i o n Ev a l u a t i o n Ev a l u a t i o n Ev a l u a t i o n Ev a l u a t i o n Ma r k e t P r o g r e s s Ev a l u a t i o n Ma r k e t P r o g r e s s Ev a l u a t i o n Ma r k e t P r o g r e s s Ev a l u a t i o n Ma r k e t P r o g r e s s Ev a l u a t i o n Ma r k e t P r o g r e s s Ev a l u a t i o n Ma r k e t P r o g r e s s Ev a l u a t i o n Ma r k e t P r o g r e s s Ev a l u a t i o n Ma r k e t P r o g r e s s Ev a l u a t i o n Sa v - Air M a r k e t T r a n s f o r m a t i o n I n i t i a t i v e No . 5/ 8 / 2 0 0 3 N E E A w e b s i t e Pa c i f i c E n e r g y A s s o c i a t e s , I n c . St e v e n S c o t t , P E Je n n i f e r S t o u t Fr e d G o r d o n Pa c i f i c E n e r g y A s s o c i a t e s , I n c . Ste v e n S c o t t , P E Fr e d G o r d o n Me t a R e s o u r c e G r o u p St e v e n S c o t t , P E Je n n i f e r S t o u t Re s e a r c h I n t o A c t i o n , I n c . Ja n e S . P e t e r s , P h D St e l l a r P r o c e s s e s , I n c . Da v e R o b i s o n , P E Re s e a r c h I n t o A c t i o n , I n c . St e l l a r P r o c e s s e s , I n c . Re s e a r c h I n t o A c t i o n , I n c . Ja n e S . P e t e r s , P h D AG . F l y n n Te c M R K T W o r k s Jo h n H . R e e d An d r e w D . O h Nic h o l a s P . H a l l Re s e a r c h I n t o A c t i o n , I n c . Ja n e S . P e t e r s , P h D Sh e l F e l d m a n M a n a g e m e n t C o n s u l t a n t s Sh e l F e l d m a n , P h D Ma r - Ma r - De c - Oc t - No v - Au g - No v - Ev a l u a t i o n Fe b - Ev a l u a t i o n Ev a l u a t i o n Ev a l u a t i o n Ev a l u a t i o n Ev a l u a t i o n Ev a l u a t i o n Ma r k e t P r o g r e s s Ev a l u a t i o n Ma r k e t P r o g r e s s Ev a l u a t i o n Ma r k e t P r o g r e s s Ev a l u a t i o n Ma r k e t P r o g r e s s Ev a l u a t i o n Ma r k e t P r o g r e s s Ev a l u a t i o n E x e c . Su m m a r y Ma r k e t P r o g r e s s Ev a l u a t i o n E x e c . Su m m a r y Ma r k e t P r o g r e s s Ev a l u a t i o n E x e c . Su m m a r y Sa v - Ai r , N o . 5/ 8 / 2 0 0 3 N E E A w e b s i t e Pa c i f i c E n e r g y A s s o c i a t e s , I n c . Da v e H e w i t t Je f f P r a t t Ga r y S m i t h Pa c i f i c E n e r g y A s s o c i a t e s , I n c . Je f f P r a t t Ga r y S m i t h Pa c i f i c E n e r g y A s s o c i a t e s Qu a n t e c , L L C Te c M R K T W o r k s Jo h n H . R e e d An d r e w D . O h Nic h o l a s P . H a l l Th e G i l m o r e R e s e a r c h G r o u p He i d i H e r m e n e t , e d . Pa c i f i c E n e r g y A s s o c i a t e s , I n c . Da v e H e w i t t Je f f P r a t t Ga r y S m i t h Ju n - Ja n - Ma r - Ap r - Ja n - Ju l - Su m m i t B l u e C o n s u l t i n g St r a t u s C o n s u l t i n g Sa v - Air , N o . 5/ 8 / 2 0 0 3 N E E A w e b s i t e Sc i e n t i f i c I r r i g a t i o n S c h e d u l i n g , N o . 5/8 / 2 0 0 3 N E E A w e b s i t e Sc i e n t i f i c I r r i g a t i o n S c h e d u l i n g , N o . 5/ 8 / 2 0 0 3 N E E A w e b s i t e Sc i e n t i f i c I r r i g a t i o n S c h e d u l i n g , N o . 5/ 8 / 2 0 0 3 N E E A w e b s i t e Sil i c o n C r y s t a l G r o w i n g F a c i l i t i e s , N o . 5/ 8 / 2 0 0 3 N E E A w e b s i t e Sil i c o n C r y s t a l G r o w i n g F a c i l i t i e s , N o . 5/ 8 / 2 0 0 3 N E E A w e b s i t e Su p e r G o o d C e n t s M a n u f a c t u r e d H o u s i n g 5/ 8 / 2 0 0 3 N E E A w e b s i t e Th e S u p e r G o o d C e n t s M a n u f a c t u r e d Ho u s i n g V e n t u r e , N o . 5/ 8 / 2 0 0 3 N E E A w e b s i t e Wa s h W i s e P r o g r a m 5/ 8 / 2 0 0 3 N E E A w e b s i t e EZ C o n s e r v e , N o . 5/ 8 / 2 0 0 3 N E E A w e b s i t e Li g h t i n g D e s i g n L a b 5/8 / 2 0 0 3 N E E A w e b s i t e Li g h t W i s e , N o . 5/ 8 / 2 0 0 3 N E E A w e b s i t e Wa s h W i s e , N o . 5/ 8 / 2 0 0 3 N E E A w e b s i t e NW A l l i a n c e R e t r o s p e c t i v e F i n a l R e p o r t Ev a l u a t i o n Ma r k e t P r o g r e s s En e r g y S t a r R e s i d e n t i a l L i g h t i n g P r o g r a m , 5/ 8 / 2 0 0 3 N E E A w e b s i t e EC O N o r t h w e s t Ju n - Ev a l u a t i o n R e p o r t No . Ev a l u a t i o n Ma r k e t P r o g r e s s No r t h w e s t E n e r g y E d u c a t i o n I n s t i t u t e 5/8 / 2 0 0 3 N E E A w e b s i t e Cu r r e n t s C o n s u l t i n g Ap r - Ev a l u a t i o n , N o . Je n n i f e r S t o u t Me t a R e s o u r c e G r o u p Ste v e n S c o t t Ev a l u a t i o n St a r t - Up P r o c e s s Li g h t i n g D e s i g n L a b 5/ 8 / 2 0 0 3 N E E A w e b s i t e Re s e a r c h I n t o A c t i o n , I n c . Ap r - Ev a l u a t i o n S u m m a r y Ja n e S . P e t e r s , P h D Ex e c u t i v e S u m m a r y F i n a l R e p o r t E x e c u t i v e Bu i l d i n g C o m m i s s i o n i n g P r a c t i c e s i n N e w 5/ 8 / 2 0 0 3 N E E A w e b s i t e SB W C o n s u l t i n g , I n c . Oc t - Su m m a r y Co n s t r u c t i o n a n d E x i s t i n g B u i l d i n g M a r k e t s W. B e c k in t h e P a c i f i c N o r t h w e s t Ka p l a n E n g i n e e r i n g Re s e a r c h Ma r k e t R e s e a r c h A C h a r a c t e r i z a t i o n o f t h e N o n r e s i d e n t i a l 5/ 8 / 2 0 0 3 N E E A w e b s i t e Ele y A s s o c i a t e s No v - Fe n e s t r a t i o n M a r k e t Re s e a r c h Ma r k e t R e s e a r c h Ba s e l i n e C h a r a c t e r i s t i c s o f t h e M u l t i - Fa m i l y 5/ 8 / 2 0 0 3 N E E A w e b s i t e Ec o t o p e De c - Se c t o r Or e g o n a n d W a s h i n g t o n Re s e a r c h Ma r k e t R e s e a r c h Ba s e l i n e C h a r a c t e r i s t i c s o f t h e N o n - 5/8 / 2 0 0 3 N E E A w e b s i t e Ec o t o p e De c - Re s i d e n t i a l S e c t o r Da v i d B a y l o n Id a h o , M o n t a n a , O r e g o n a n d W a s h i n g t o n Mik e K e n n e d y Sh e l l y B o r r e l l i Re s e a r c h Ma r k e t R e s e a r c h Ba s e l i n e C h a r a c t e r i s t i c s o f t h e R e s i d e n t i a l 5/8 / 2 0 0 3 N E E A w e b s i t e Ec o t o p e De c - Se c t o r Da v i d B a y l o n Id a h o , M o n t a n a , O r e g o n a n d W a s h i n g t o n Sh e l l y B o r r e l l i Mic h a e l K e n n e d y Re s e a r c h Ma r k e t R e s e a r c h Co m m e r c i a l a n d I n d u s t r i a l L i g h t i n g S t u d y , 5/ 8 / 2 0 0 3 N E E A w e b s i t e XE N E R G Y I n c . De c - Vo l u m e 1 Ri s i n g S u n E n t e r p r i s e s Pa c i f i c E n e r g y A s s o c i a t e s En e r g y M a r k e t I n n o v a t i o n s Re s e a r c h Ma r k e t R e s e a r c h Co m m e r c i a l B u i l d i n g s I n i t i a t i v e 5/ 8 / 2 0 0 3 N E E A w e b s i t e SC H I C K C O N S U L T I N G Se p - O 2 Ta r g e t M a r k e t P r i o r i t i e s Ha r o l d ( S k i p ) S c h i c k Pa c i f i c E n e r g y A s s o c i a t e s , I n c . Le s T u m i d a j Re s e a r c h Ma r k e t R e s e a r c h En e r g y E f f i c i e n c y w i t h i n t h e P u l p a n d 5/ 8 / 2 0 0 3 N E E A w e b s i t e Du c k e r W o r l d w i d e No v - Pa p e r , W a t e r a n d W a s t e w a t e r a n d Ir r i g a t i o n M a r k e t s i n t h e P a c i f i c N o r t h w e s t Re s e a r c h Ma r k e t R e s e a r c h En e r g y E f f i c i e n t L i g h t i n g i n N e w 5/8 / 2 0 0 3 N E E A w e b s i t e Ec o s C o n s u l t i n g Ma y - Co n s t r u c t i o n Be n y a L i g h t i n g D e s i g n Ris i n g S u n E n t e r p r i s e s Re s e a r c h Ma r k e t R e s e a r c h Ma r k e t A s s e s s m e n t o f t h e I n d e p e n d e n t l y 5/8 / 2 0 0 3 N E E A w e b s i t e Qu a n t u m C o n s u l t i n g I n c . De c - Ow n e d R e t a i l F o o d S e c t o r i n t h e P a c i f i c No r t h w e s t Re s e a r c h Ma r k e t R e s e a r c h Ne w C o m m e r c i a l O f f i c e B u i l d i n g s : 5/ 8 / 2 0 0 3 N E E A w e b s i t e Wa s h i n g t o n S t a t e U n i v e r s i t y Se p - De v e l o p i n g S t r a t e g i c M a r k e t Ri c k K u n k l e , P E Tr a n s f o r m a t i o n I n i t i a t i v e s f o r E n e r g y Lo r e n L u t z e n h i s e r , P h D Ef f i c i e n C y Re s e a r c h Ma r k e t R e s e a r c h Op p o r t u n i t i e s f o r N e w A p p l i a n c e M a r k e t 5/ 8 / 2 0 0 3 N E E A w e b s i t e Fo u r W i n d s A l l i a n c e Au g - Tr a n s f o r m a t i o n P r o g r a m s i n t h e P a c i f i c D& R I n t e r n a t i o n a l No r t h w e s t Re s e a r c h Ma r k e t R e s e a r c h Pa c i f i c N o r t h w e s t W a t e r a n d W a s t e w a t e r 5/ 8 / 2 0 0 3 N E E A w e b s i t e Qu a n t u m C o n s u l t i n g I n c . Ma y - Ma r k e t A s s e s s m e n t Ad o l f s o n A s s o c i a t e s I n c . Su m m i t B l u e C o n s u l t i n g NW A l l i a n c e R e t r o s p e c t i v e F i n a l R e p o r t St r a t u s C o n s u l t i n g Re s e a r c h Ma r k e t R e s e a r c h Re g i o n a l P u b l i c I n f o r m a t i o n P r o g r a m f o r 5/ 8 / 2 0 0 3 NE E A w e b s i t e Co l e & W e b e r Ju l - O O th e E f f i c i e n t B u i l d i n g P r a c t i c e s I n i t i a t i v e Re s e a r c h Ma r k e t R e s e a r c h Re s i d e n t i a l E n e r g y E f f i c i e n t L i g h t i n g 5/ 8 / 2 0 0 3 NE E A w e b s i t e Re g i o n a l E c o n o m i c R e s e a r c h , I n c . Ap r - Co n s u m e r R e s e a r c h Re s e a r c h Ma r k e t R e s e a r c h Su m m a r y o f a W o r k s h o p o n t h e F u t u r e o f 5/8 / 2 0 0 3 NE E A w e b s i t e No r t h w e s t E l ' 8 r g y E f f i c i e n c y A l l i a n c e Se p - El e c t r i c E n e r g y U s e Be n B r o n f m a n Re s e a r c h Ma r k e t R e s e a r c h Va r i a b l e F r e q u e n c y D r i v e s 5/ 8 / 2 0 0 3 NE E A w e b s i t e Ea s t o n C o n s u l t a n t s , I n c . Ju n - O O Re v i e w Re v i e w o f L i t e r a t u r e 1 9 8 9 - Im p a c t s o f F o r c e d A i r D i s t r i b u t i o n S y s t e m s 5/ 8 / 2 0 0 3 NE E A w e b s i t e OS U E x t e n s i o n E n e r g y P r o g r a m Ja n - 9 8 19 9 7 on H o m e s a n d P o t e n t i a l f o r I m p r o v e m e n t s B. B o e Sp e c i a l R e p o r t Sp e c i a l R e p o r t Pr o d u c t T e s t i n g : M a g n a D r i v e , R e p o r t N o . 5/ 8 / 2 0 0 3 N E E A we b s i t e Mo t o r S y s t e m s . R e s o u r c e F a c i l i t y Ma r - (M S R F ) Or A " " n S t " . . . I I n i v A r " i t v Sp e c i a l R e p o r t Sp e c i a l R e p o r 1 - E n h a n c e d Pu b l i c B u i l d i n g C o m m i s s i o n i n g i n t h e 5/ 8 / 2 0 0 3 NE E A w e b s i t e Qu a n t u m C o n s u l t i n g , I n c . Ju l - 9 9 Ba s e l i n e Pa c i f i c N o r t h w e s t , N o . Ph i l W i l l e m s Sp e c i a l R e p o r t Sp e c i a l R e p o r t P r o d u c t En v i n t a O n e - 2 - Fi v e P i l o t 5/ 8 / 2 0 0 3 NE E A w e b s i t e Re s e a r c h I n t o A c t i o n , I n c . Ma r - As s e s s m e n t As s e s s m e n t Ja n e S . P e t e r s , P h D Sp e c i a l R e p o r t Or g a n i z a t i o n a l S t r u c t u r e Li g h t i n g D e s i g n L a b 5/ 8 / 2 0 0 3 NE E A w e b s i t e He i n C o n s u l t i n g G r o u p Ju n - O O Re v i e w a n d Re v i e w a n d Ala n n a E . H e i n Re c o m m e n d a t i o n s Re c o m m e n d a t i o n s St u d y Ca s e S t u d y Gr e e n C i t y B u i l d i n g s A p p l y i n g t h e L E E D 5/ 8 / 2 0 0 3 NE E A w e b s i t e XE N E R G Y I n c . Oc t - Ra t i n g S y s t e m SE R A A r c h i t e c t s St u d y a n d Ba s e l i n e S t u d y a n d En e r g y S t a r H i g h E f f i c i e n c y R e s i d e n t i a l 5/ 8 / 2 0 0 3 NE E A w e b s i t e Ma c r o I n t e r n a t i o n a l , I n c . De o - 9 8 Ch a r a c t e r i z a t i o n Ma r k e t C h a r a c t e r i z a t i o n Win d o w s St a f f m e m o Po r t f o l i o D e v e l o p m e n t Su m m a r y o f O c t o b e r R e t r e a t 6/ 1 2 / 2 0 0 3 on - 6 i t e Da r l e n e N e m n i c h , P o r t f o l i o C o m m i t t e e Oc t - Co m m i t t e e Ch a i r ( a t t h e t i m e ) MT S t r a t e g y Co m m e r c i a l B u i l d i n g s CB I I B e t t e r B r i c k s 2 0 0 3 P r o p o s e d A c t i v i t i e s 6/1 2 / 2 0 0 3 on - 6 i t e Je f f H a r r i s & D a v e H e w i t t , o t h e r s ? Oc t - In i t i a t i v e an d B u d g e t me m o in t e m a l Bo a r d C o m m i t t e e C h a r t e r s 6/1 2 / 2 0 0 3 on - 6 i t e po w e r p o i n t Ho w B u s i n e s s i s A c c o m p l i s h e d 6/ 1 2 / 2 0 0 3 on - 6 i t e Je f f H a r r i s Ju n - 0 2 Sta f f in t e m a l Ut i l i t y D i s t r i b u t i o n S y s t e m E f f i c i e n c y 6/ 1 2 / 2 0 0 3 on - 6 i t e , S u s a n Je f f H a r r i s , o t h e r s s Ja n - 0 3 re c o m m e n d a t i o n In i t i a t i v e He n n e n e t me m o me m o Po r t f o l i o D e v e l o p m e n t Po r t f o l i o D e v e l o p m e n t C o m m i t t e e A g e n d a 6/ 1 2 / 2 0 0 3 on - 6 i t e , J e f f Je f f H a r r i s , o t h e r s Ja n - 0 3 Co m m i t t e e an d b a c k u p m a t e r i a l s Ha r r i s re p o r t Lig h t i n g d a t a Vo l u m e 1 : N a t i o n a l L i g h t i n g I n v e n t o r y a n d 6/ 6 / 2 0 0 3 do w n l o a d f r o m Na v i g a n t C o n s u l t i n g - f o r D O E / E E R E Se p - En e r g y C o n s u m p t i o n E s t i m a t e we b co n f i d e n t i a l dr a f t r e p o r t CF L P r o d u c t S a l e s D a t a - R e p o r t f o r 6/ 1 9 / 2 0 0 3 Ma r c i S a n d e r s co n t r a c t o r ( E c o s ? ) Ju n - 0 3 Ja n u a r y - Ma r c h 2 0 0 3 re s e a r c h Ma r k e t R e s e a r c h CB I B a s e l i n e : S c h o o l D i s t r i c t s P l a n n i n g 7/3 0 / 2 0 0 3 Li s S a u n d e r s Ju l - 0 3 Co n s t r u c t i o n ut i l i t y r e p o r t ev a l u a t i o n Co n s e r v a t i o n K i t P r o g r a m E v a l u a t i o n : 8/2 6 / 2 0 0 3 Se a t t l e C i t y L i g h t De b r a T a c h i b a n a , S e a t t l e C i t y L i g h t Ma ~ 3 Tr a n s f o r m i n g t h e R e s i d e n t i a l U s e o f C F L s re p o r t pr o j e c t d e s c r i p t i o n Re s i d e n t i a l S e c t o r I n i t i a t i v e - D R A F T 8/ 2 6 / 2 0 0 3 All i a n c e s t a f f Au g - Pro j e c t D e s c r i p t i o n C o n s u m e r P r o d u c t s sp r e a d s h e e t MA R n u m b e r s All i a n c e P r o g r e s s T r a c k e r ( Y e a r 2 0 0 2 - 9/ 1 0 / 2 0 0 3 Je f f H a r r i s Je f f H a r r i s , S u s a n H e r m e n e t Ap r - Ba s e l i n e ) sp r e a d s h e e t NW P P C b a s e l i n e s NW P P C S p a c e C o n d i t i o n a i n g S y s t e m a n d 9/ 1 0 / 2 0 0 3 To m E c k m a n To m E c k m a n , N W P P C Fu e l b y H o u s i n g T y p e & V i n t a g e Su m m i t B l u e C o n s u l t i n g St r a t u s C o n s u l t i n g NW A l l i a n c e R e t r o s p e c t i v e F i n a l R e p o r t Alliance Project Activity Avista -Idaho Territory February 2004 This document details specific Northwest Energy Efficiency Alliance (Alliance) program activity that has occurred in territory covered by Avista in the state of Idaho over the last three years. The information is organized by sector and then by each project. Because the markets for energy efficient products and services are not limited to state boundaries, in some cases we have included program activity that has occurred in the Spokane area knowing that these benefit consumers who live in Northern Idaho but shop and do business is Spokane. RESIDENTIAL PROGRAMS ENERGY STAR(li,) Home Products Program Program Retailers There are (37) participating retailers that customers of Avista Idaho visit including (5) Sears , (4) Home Depots, (3) Lowe , (2) Best Buys, (2) Costcos and (21) independent stores. Backwoods Solar Bellwood Bonners Ferry Appliance/Collective Dreams Cope s Appliance Club Deranleau s Appl. and TV Deranleau Fred's Appliance Furniture Exchang Hoene Hardware Home Appliance and Electronics Home Depot #1803 Home Depot #1808 Howard Hughes Appliance Hughes Home Center Kluss Appliance Largent's Lindsley s Furniture & Appliance Lowe of Coeur d' Alene Brien Inc. NW Appliance Rex TV Sandpoint Sandpoint Bonners Ferry Coeur d'Alene Moscow Lewiston Coeur d'Alene Kellogg Cottonwood Sandpoint Coeur d'Alene Lewiston Moscow St. Maries Runge Furniture Lewiston Lewiston Grangeville Coeur d'Alene Priest River Lewiston Coeur d'Alene Sears #2209 Lewiston Sears #2349 Coeur d'Alene Sears #3071-ind Sand point Simon Furniture Orofino Best Buy 355 Spokane Best Buy 362 Spokane Costco II Spokane Costco III Spokane Fred's Appliance Spokane Home Depot #4714 Spokane Home Depot #4714 Spokane Lowe s of Spokane Spokane Lowe s of Spokane Valley Spokane Sears #1029 Spokane Sears #1038 Spokane The Kitchen Place Spokane University Appliance & TV Inc.Spokane Retailer Cooperative Marketing Fund Advertising In 2001 and 2002 , the Program offered cooperative marketing funds that drove retailers to develop their own marketing events, supported by advertising and in- store elements around ENERGY STAR. In 2001 , cooperative marketing projects were implemented between September 15th and February 28th . In 2002 , projectswere implemented between July 31st and December 31st. In the Avista area , thefollowing retailers took advantage of the cooperative funds during those two offerings: . Fred's Appliance, Coeur d'Alene & Spokane . Home Appliance, Sand point Howard Hughes, Moscow University Appliance , Spokane Furniture Exchange , Kellogg Fred's Arwliance - 2001, 2002 and 2003 (throuqh Double Your SavinQ~ Fred's participated in the popular Spokane "Taste of Homes" event, arranging for placement of ENERGY STAR qualified appliances on stage during the event with prominent placement of the ENERGY STAR logo. They also featured a drawing in both their Spokane and Coeur d'Alene stores for an ENERGY STAR qualified dishwasher, and they funded 6 months of utility bill payments for the winner. They supported their "ENERGY STAR Days " event with print and radio advertising, including a 2-page editorial on the benefits of ENERGY STAR qualified appliances. Home A(2pliance - 2001 Home Appliance furthered ENERGY STAR awareness in several ways: They set up a display of qualified appliances in the front window for 60 days with a large ENERGY STAR logo hanging overhead. The display featured an ENERGY STAR qualified Bosch dishwasher that customers could enter to win. They offered free delivery when customers purchased an ENERGY STAR qualified product, an additional offer unique to the event. Print and radio advertising supported the event and a local radio station was on-site for live remotes at the kick-off and for the drawing of the winner. To further entice customers to come attend the event, a drawing for prizes was held , along with live music and a barbeque. Howard Huqhes - 2002 Howard Hughes ' " ENERGY STAR Days" event had several elements and made use of interactive marketing materials to further ENERGY STAR brand awareness. During the first month of the promotion, each customer who filled out an ENERGY STAR quiz received a free ENERGY STAR qualified CFL. For the duration of the event, employees asked questions about the benefits of qualified products, and customers who found the answers by reading Program marketing materials placed on a display of qualified clothes washers could enter to win qualified clothes washer. The clothes washer display had a custom sign placed over the top that featured the ENERGY STAR logo. Additionally, any customer who purchased an ENERGY STAR qualified product during the promotional period received half- price delivery. A reader board, located on the busiest highway in the area, announced "ENERGY STAR Days - Enter to win an ENERGY STAR qualified clothes washer. University AQ,Pliance - 2002 University Appliance created an "ENERGY STAR Showcase" that featured 12 pairs of qualified clothes washers with an awning, lighting and ENERGY STAR signage. The awning featured the ENERGY STAR logo on all four corners with "ENERGY STAR Showcase" in the middle. The clothes washers featured in the showcase were 'live' for demonstrations , and midway through the promotion qualified refrigerators and dishwashers were added. To promote the showcase, University Appliance utilized television and print advertising. Furniture Exchanqe - 2002 To illustrate the variety of products on which the label can be found, the store placed a grouping of ENERGY STAR qualified appliances in the front window. To further long-term brand awareness, a neon sign in the shape of an ENERGY STAR logo was placed over the appliance grouping. Field Services Field services continue to benefit Avista in procuring retail participation in the Program s promotions and events. Field staff works closely with retailers to train retail staff, provide POP materials , introduce promotions such as Double Your Savings with ENERGY 8T AR and (gJHome for the Holidays and to persuade retailers to become active in outreach events and advertising. The field representatives are seen as the most valuable component of the program by most retail partners, who rely on field representatives' regular store visits, utility incentive lists , qualified model lists and updated training materials. Additionally, field representatives provide opportunities for retailers to display products at outreach events and participate in utility and program coordinated promotions. Marketing Promotion Point of Purchase Materials New tools released in 2002-2003 included large refrigerator magnets and slide-rule calculators for all appliances. These tools help consumers to calculate their individual savings when upgrading to ENERGY 8T AR qualified appliances. In addition to these tools, some standard point-of-purchase materials have been redesigned to incorporate the new ENERGY 8T AR logo launched the first half of 2003. These materials were distributed to retailers. ;Ef~, :~~ i 4:8\' ;:. . . . ~ )'tiT$, i~II:31!lj;~(.~!ltrp) . . (~~:~JijTlit~DrR~J~J:~rti.:r~? . . . . ~;h:fiii~,'Ir",' (p, : . j!l!ti!l'A~\I!I'~,"uc~). Year refrigerator KWhIyr Yearly 10 Yearpurchased saved I18vi"gs. I18ving. " . 1982 : .;. , 689 ,$48.23;:$482.3D.. . . . 1984 .. .. 837 54459 . S445:SO :1986:. 572 540.. $400;40 . 1988 '. . 462 $32~ $mAD ;1890 ",;;. . il1~::$28.98 : 5289.80. . . 1992 .319 .$22.33 .5223. . . 19901:;:151' :51 0~75105.70.1996 159 $11.13 5111;30 . ' BaSed cin $0.07 ptirkWh ' . . ::.: / =:/::r~~::: Marketing Promotion Consumer Outreach Events To date, the ENERGY 8TAR Home Products Program has participated in 15 outreach events to which Avista s Idaho customers were exposed. Outreach events ranged from home and garden shows to the retail-sponsored in-store events. Marketing Promotion Public Relations The Program has worked with media throughout the Northwest to secure placement of press releases, television coverage of events and radio PSA announcements. Coverage to which Avista Idaho customers were exposed includes six newspaper articles , nine TV appearances and two radio announcements. The Program also has created a variety of canned articles. Canned articles areavailable to media and utility partners in short and long versions. They have finalized layout with headlines and graphics to make placement easy in newspapers, newsletters and bill inserts. Radio Remotes KCRK Colville 2002/02/23 KCRK Bonner s Ferry 2002/02/23 Marketing Promotion 2004 Double Your Savings with ENERGY STAR From April 15th through July 15th of 2002, the 13-region collaborative effort of utilities regional and state organizations , retailers and manufacturers provided over 66 000 rebates on ENERGY STAR qualified clothes washers under the promotion title Double Your Savings with ENERGY STAR." The Program brought together partners and 434 retailers, resulting in a distinct rise in market share for qualified clothes washers in the Northwest. As a result of the overwhelming success of last year s promotion , national ENERGY STAR is launching the promotion again for the same time period this spring 2004. This year, the promotion is designed to encourage customers to save even more energy and money on utility bills than last year by choosing models with the highest efficiency ratings. Participating utility partners will distribute rebate forms via bill stuffers , newsletters, newspapers or direct mail, offering a minimum $25 rebate off the purchase price of ENERGY STAR qualified clothes washers with a Modified Energy Factor (MEF) of 1.42-, and a minimum $50 rebate for 1.6 and above MEF. Rebate forms will feature an additional $25 or $50 rebate from participating manufacturers based on the same MEF criteria. This rebate structure provides customers a wide array of product choices as well as a minimum combined incentive of $50. To date , 53 utilities have signed up to participate, including Avista who will distribute rebate forms via bill stuffers or a newsletter to their Idaho customers. Promotion Elements: Promotion Timeline: April 15 through July 15, 2004 Rebate Amounts:Avista: MEF 1.42-59 = $25 MEF 1.6 or greater = $50 Manufacturers: MEF 1.42-59 = $25 MEF 1.6 or greater - $50 Coupon Distribution: Coupons will be distributed to Avista s Idaho residential customers through bill stuffers or newsletters. Only electric customers Idaho are eligible. Coupon Redemption: Customers will mail rebate forms to the Program for verification. Customer Payment: Manufacturer and Avista rebate will be coordinated through the Program. Retailer & Utility Support: The Program will not only provide field support to retailers , driving their participation through training sessions , in-store placement of POP and prominent display of qualified products, but also will keep Avista updated and offer support at community outreach events. Retailers will also be offered cooperative advertising funding. Retailers will receive additional training to explain the short-term changes in Avista program. . . Marketing Promotion ENERGY STAR cmHome for the Holidays For the 2003 Holiday Season , the Program offered utility partners, including Avista and retail partners a chance to participate in an ENERGY STAR consumer awareness campaign , creating excitement around the ENERGY STAR label by offering an online Sweepstakes (www.athomefortheholidavs.net) open to all Northwest consumers. The Sweepstakes ran from November 28th through December 26th, 2003 and offered great ENERGY STAR qualified prizes: Whirlpool trio of ENERGY STAR qualified appliances, including a refrigerator dishwasher and clothes washer, valued at $2 230. Panasonic 37" Plasma Screen TV, valued at $4 000. 10 Panasonic DVD Players Retail partners were given the opportunity to participate in an additional component in which they distributed in-store instant Scratch & Win game cards where customers scratched to win "cool ENERGY STAR gear." In return , retailers committed to advertising promoting the game cards and sweepstakes. All retailers whether or not they signed up for the additional scratch & win component, received easel back cards and door clings to help them in promoting the sweepstakes further. The following retailers, shopped by Idaho Avista customers , signed on for the opportunity to offer their customers the additional scratch and win component: The Program worked with utility partners throughout the Northwest, including Avista to promote the sweepstakes to their customers. Partners were encouraged to alert their customers to the sweepstakes opportunity through web placement, lobby displays, newsletter articles, print ads and other venues. Marketing Promotion Make Change to ENERGY STAR The Make a Change to ENERGY STAR Toolkit was offered to Alliance partners in May 2002. The toolkit offered simple planning steps, templates, checklists , timelines and other suggestions for implementing Make a Change to ENERGY STAR. The Make a Change to ENERGY STAR promotion was piloted in four cities in the region last year including Spokane, W A. The media coverage generated was certain to penetrate into Idaho. The concept behind this promotion was to partner with high- profile organizations and facilities in individual communities to conduct home product change-outs , providing an avenue for spotlighting the increased energy efficiency and money savings that result from switching to ENERGY STAR qualified products. The promotion benefits utility partners in many ways, including: Strengthening community ties and establishing community partners Generating sustained kWh savings and demonstrating utility partner commitment to building a better community for your customers Enhancing relationships with appliance manufacturers and local retailers Driving public awareness for energy efficiency, visibly aligning promotional partners and encourages consumers to take action to replace their current less- efficient home products With the Make a Change to ENERGY STAR Toolkit, utility partners have the ability to leverage the Program s expertise in designing and executing this promotion. ENERGY STAR Market Penetration Rates for Idaho Market penetration rates for the state of Idaho continue to climb. Sales of ENERGY STAR qualified clothes washers, dishwashers and refrigerators have steadily risen. The dishwasher market in quarter one of 2003 saw a dramatic climb. 2001 - Market Penetration Results for Idaho 2002 - Market Penetration Results for Idaho 2003 - Market Penetration Results for Idaho st Qtr 2:o~a;t)~t 3rd Qtr ENERGY STAR Residential Lighting Program Program Lighting Retailer in A vista Idaho territory Retailer Address City CO-OP Supply O. Box 1709 Coeur d' Alene Fred Meyer 560 West Kathleen Coeur d' Alene Home Depot 220 W. Kathleen Ave.Coeur d' Alene Lowe 901 Appleway Ave.Coeur d' Alene Rite Aid Corporation 208 West Ironwood Square Coeur d' Alene Simons Hardware Do-It-Center 1217 N 4th St Coeur d' Alene D & B Farm & Home Stores 170 E. Kathleen Coeur d'Alene Inc. Hoene Hardware 419 Main O. Box 78 Cottonwood Kamiah True Value 403 Main Street P.O. Box 368 Kamiah Kellogg Lumber Company 202 West Station Ave.Kellogg Sunny Side Drug Ace 131 W Cameron Ave Kellogg Trustworthy Hardware and Furni 101 N Hill St Kellogg Kendrick Hardware 701 E. Main St.Kendrick Rosauer s Supermarkets Inc.332 Thain Rd.Lewiston Thurman Supply 1715 Idaho St.Lewiston JJ Building Supplies Inc 123 W 7th St Moscow Moscow & Pullman Building Supply 705 N. Main Moscow Rite Aid Corporation 1810 West Pullman Rd.Moscow Spence Hardware & Supply 915 White Ave Moscow Tidyman s Inc #15 S 1638 Blaine Moscow Tri-State Distributors 1104 Pullman Rd.Moscow Winco Foods 1700 W. Pullman Rd.Moscow Orofino s Building Supply 165 Riverside Ave Orofino Plummer & Wagner 412 Second St South Hwy 95 Potlatch Backwoods Solar 1395 Rolling Thunder Ridge Sandpoint Co-op Ace 125 Tibbetts Lane Sand point Merwin s Hardware 201 N 3rd Ave Sandpoint Sand point Drug Ace 602 N. 5th Sandpoint Cooperative Marketing Cooperative marketing is cost-shared funding which supports retailer s advertising containing ENERGY STAR messaging and outreach events. The Alliance Cooperative Marketing Fund provides matching dollars available to help utilities, retailers and lighting manufacturers to promote the sale of ENERGY STAR lighting products. Funding is targeted at comprehensive, retail-based special projects that include prominent in-store placement of qualifying products, ENERGY STAR messaging and advertising. The Alliance distributes funds on a first-come, first-served basis. The Program Marketing team provided assistance to the Cooperative Marketing Fund in 2002 by creating end-cap displays. These end-caps, known as Header Boards, were giant versions of the ENERGY STAR logo and were very successful. Since their creation , these displays have landed in many retail locations throughout Avista territory. Money Isnt All You re Saving Retailer Cooperative Marketing Fund Advertising Coop Supply promoted ENERGY STAR bulbs through a series of radio and newsprint advertisements. ENERGY STAR end-caps and in-store point-of-purchase materials were used to support this promotion. Radio ads on KVNI - AM and print ads ran for a total of six months starting in January 2002. Idaho Lights , a retailer in Coeur d'Alene, 10 as well as Evergreen Lighting in Spokane, W A, ran 20- 60 second announcements in May of 2002 that aired on KXL Y- AM / KXL Y- FM , a local Idaho and NE WA radio station. The ads promoted saving energy by purchasing ENERGY STAR energy efficient products such as fixtures, ceiling fans, and bulbs. Hoene Hardware in Cottonwood ran a series of six CFL ads in the Cottonwood Chronicle and the Granville Press in November 2002. Hoene Hardware in Cottonwood featured ENERGY STAR qualified torchieres in its holiday ad. The ad ran in four newspapers in 2002: The Idaho County Press December 18th The Shopper on December 19th The Lewis County Herald December 19th and The Cottonwood Chronicle on December 19th Merwin s True Value , in Sand point , participated in training on ENERGY STAR CFLs and product knowledge on May 5, 2003. Attendance included the new electrical manager and several staff members. Field coordinators loaned the store a lighting display box and demonstrated various CFLs and compared the three basic bulbs- 15, 20 and 25 watt. Field Services Field coordinators facilitated training to retail staff and assisted with Program Point- of Purchase (POP) good placement, such as shelf-talkers, isle wobblers, bulb- wheels, brochures , and other POP materials. Field coordinators also managed outreach events educating consumers about ENERGY STAR and the program. In addition to their work with retailers, field coordinators regularly visit utilities to provide Program updates, encourage linkages with local retailers , provide ENERGY STAR product training, and support utility-driven consumer awareness activities. Another key function performed by the field coordinators is the collection of retail level data on market indicators , such as product variety and price. Finally, field coordinators track changes in ENERGY STAR lighting sales in specific utility territories, watching for opportunities to leverage local successes and bolster areas with lower than average sales. Field representatives coordinated periodic visits with Avista staff to stay abreast of utility initiatives and to provide Program updates whenever possible. Field coordinators has also relayed developments at local retailers to the utility. Retailer Handbook Distribution Avista retailers who are a part of the Program were provided with a Retail Handbook in 2002 and a new updated Retailer Handbook in 2003. Each Handbook contained in-depth information on how to fully utilize the retail benefits of the Program including marketing and promotions linkage, details of field coordinator support, and how to apply for co-op funding. These miniature handbooks pack in the information yet are designed small enough to fit into a retail apron pocket. These books were delivered to retailers by the program s field representatives. Outreach/ Special Promotions Idaho Building Contractors Home and Garden Show Coeur d' Alene, March 2002 The Program participated in the Idaho Building-Contractors Home and Garden Show in March 2002. Field coordinators spoke with show attendees about ENERGY STAR lighting and provided materials explaining CFL usage and savings. Torchiere lamp turn- November 2002 The Program marketing team worked in conjunction with the field staff to create support materials for a torchiere lamp turn-in held in Lewiston and Moscow, Idaho, on November 16, 2002. Newspaper advertisements were created and placed on behalf of the participating retailers, and a coupon was designed and distributed by the Program as well. Multiple media outlets showed up for the event. When combined with another event in nearby Pullman, Washington , over 325 halogen torchieres were recycled. Lewiston Clarkston Home & Garden Show Lewiston , March 2002 Field coordinators staffed the Lewiston Clarkston Home & Garden Show. The booth was setup with ENERGY STAR lighting displays. A local retailer loaned the Program a front-loader and top loader ENERGY STAR qualified washers for display. The event provided a great venue to further build relations with the public and partners. The show offered 500 cotton tote bags to people attending the home show. The Program arranged to have the ENERGY STAR logo printed on the bag and had the bags stuffed with lighting brochures (premiums were reserved for booth visitors). The event was also good for feedback regarding CFL performance, and the local Zero Energy Home project at the Nez Perce Tribal Fish Hatchery. North Coast Electric ENERGY STAR Seminar Coeur d' Alene, October2003 Field coordinators attended the fourth North Coast Electric Seminar and lunch, this time held at the Coeur d' Alene branch. The event included exhibits from the Program, manufacturers, utilities and the store itself. It also provided a great opportunity to connect with builders and vendors while providing support to the distributor. Field staff reviewed ENERGY STAR materials with Chris Drake and Mike Littrel from Avista , with emphasis on Lightingplans.com and the commercial section Energystar.gov. Representatives from TCP , Nutone, Columbia Lighting and Sylvania were also at the event to display their products. Lowe Coeur d' Alene - November 2003 Field staff conducted an extended visit to this Lowe s store in November. Training with the department staff on ENERGY STAR products took place. Marketing Materials Marketing has been key for creating a successful ENERGY STAR Residential Lighting Program , which includes cooperative marketing, retail-based customer outreach events, marketing products , and public relations. The web site !ightsite.net has proven to be a valuable marketing tool. Not only does the site offer indepth information on the subject energy efficient lighting, but it also provides some useful programmatic tools. The Data Center is a good example of how utilities can access Program services and specific information. Retailer /ocalor at www.//ghts/te.netNorthern Lights Every two months , the Program releases Northern Lights an industry newsletter primarily written for retailers. Each newsletter features success stories and the latest retail promotional information in quick-reading, double-sided issue distributed to over 000 contacts. All of the cooperative marketing fund partners are eventually featured to highlight their individual promotions and stories. Northern Lights also contain product features , partner interviews, event photos , and Program contact information. ~QIW~(n' ~ights g~. ~i-:fEl ~€?;:~~~. ".-- '"'~" ;;IT ;oJ ._-~._-"-- "'~. .,. .. :J -'"'~._-'--' ..~-, 2"2.'~;':::'::E';:' =;;.,:..._,"-""........~,..."..~.".- "~._""'-'~"._..,_......_~"'.- -. ~ ,~" ..,." '--.."..~._..__.- -"'._"-'-"-'"",-~""",,,-,- .,--. n.,-.-...-h' -.-.--..-..- ...._._.~.._~. ~~'.:;;::E==:- ::~: ~~;~~'i:-' "";::-:-'"",-;,.....= ~t~~~. ~~;~::;;::.::;':';:-~",":~ ~~f~~f.:; Sold On CFLs BRIGHT !DEnS """""'.".""""'. ..-.----. -, ._,-."-,..,~.~:~'"~ ':;'. '::.:.,,:, :'."::'=-.:.:.::::::::::=-",:;,,-_.~".,-._~ "...,.-...,..,-"... =~;~7"..;;"':;:~= =::':::' :'::;';:::::':::::':""""'~'--"'-"" -""_."-".".,'::::':;;~;~';"~':;';,~:-;;; ::::":.~':.":.::..:.-::;: ~.;~!~ ~~~ '""". "__'0..." -........--.-., :;:, =-:~=-L:':= ~.:~:::. ~~~ ;-;~'::~..::::~::~~?;;j. ;;:i::~ :::;~-:-2;:; .....-==;.-=- Data Center Rudy To Seove '....,....."-",......~",.._- A.~'~~:.='2 ~::'::';:"7.;7=2 ...~,,~,_.-.~..... _._..~...~'....-_...~""...-.~,_. ,......~,.,-"...._---_'_'.~-,._.~. ....".~'._,-~,,~.;:::,:.:::--"'-_.~. ::"',":,:':'::::';:~':':;,"':::.:". 7;':-::;;" : ;:;::.:-::";;,:".;';:~;:"::,- ,'--.~. ..."..~, ..... ~.~ ",.,. . ..",._.......~~_.~,.- .-.."-,.-.,",..,,, _._..",.,~-...",", --_'-'0'_""' - ==.:..:.~~;:~:,~~'~:: ~::":::,,':;:~:~;'~;;,::::._"._...~.~- -..-...- eBulietin The online eBulietin is distributed to a comprehensive list of utilities, industry contacts and lighting manufacturers. CFL Disposal Fast Facts The Compact Florescent Lamps Disposal Fast Fact sheet was finalized and printed. This double-sided sheet was created to explain the essential facts on CFLs, mercury, and disposal and provides graphs and statistics on the occurrences of mercury in common household products. Utilities, industry partners, and retailers received the fact sheet upon request or during scheduled utility visits by the Program field staff. It is also be available for download from !ightsite.net. CFL Disposal Facts Tear Pad Similar to the CFL disposal fast facts sheet, this double-sided tear pad is intended to quickly convey disposal options and facts about CFLs for retail staff and consumers. These 4x4 inch pads contain 100 sheets each , and can easily be positioned at the point-of-sale for consumer access or used as a retail-training piece. The pads have been distributed to the Program field coordinators, which were delivered to retailers. Bulb Wheel To complement the existing Smart Choice materials - shelf talker, brochure, and aisle wobblers - the Program refreshed the popular bulb selection wheel with Smart Choice theme colors and new savings data and lumen information. Retail distribution was through the field staff. CFL Value and Benefits Tear Pad This marketing piece is another quick-reference sheet for consumers and retailers to learn about the values and benefits of using ENERGY STAR qualified lighting products. The double-sided tear sheet contains a savings table and bulb application information. Auto-up Display Essentially an upright, vertical banner, the auto-up is intended for use at any and all consumer outreach and Program events. The display highlights the facts that ENERGY STAR qualified lighting uses up to 750/0 less electricity and last 6 - 10 times longer than traditional lighting. Retailer Handbook (Pocket Guide) The newly revised guide incorporates substantial new information in to help retailand lighting showroom staff find the right light for the correct application for customers. This includes more specifics on lumen output, wattage comparisons and savings information , as well as color rendering and color temperature specifics. The new size is more manageable at 3x4 to fit in the pockets of aprons used by many retail staff. CFL Disposal Kit The CFL Disposal Kit is a response to utility requests for assistance with the issue of mercury in the bulbs and proper disposal. The Program completed the CFL Disposal Kit in November 2003 and made an initial announcement of the Kit's availability in the eBulietin. Subsequent revisions have been made, and kits were made available via !i.ghtsite.net. 2002 Marketing Materials shelf-talker, pens, pins, magnets, bulb-wheel, lighting brochure, stickers and product tag 2003 Marketing Materials . .."""'~""""" bulb-wheel, brochure, fact sheet about CFL tear sheet, wobbler, lighting brochure, ENERGY STARpen, shelf-talker, retailer pocket guide, magnet. Light Box Displays and Smart Choice brochures were use for Earth Day activities in 2003 at Washington retailers and outreach events. Earth Day Posters were announced in the March 2003 eBulietin. Field coordinators distributed posters to many retailers and at many outreach events. Product Sales CFL sales in Idaho for 2002 and the first three quarters of 2003 were impressive totaling 584 968. The numbers for fourth quarter are not yet available. Regional CFL Sales: IDAHO CFL Sales in Idaho First 3 Quarters 120 000 100 000 000 000 000 000 COM M E RCIAL PROGRAMS BetterBricks This project serves as a resource for information and services to help architects developers , builders and facilities managers incorporate energy efficiency into their building designs, construction plans and day-to-day maintenance operations. Although the program has screening criteria, generally information provided by BetterBricks is free and services are provided at little or no cost to Northwest building professionals. A number of program components may have influenced market actors working in Avista territory in Northern Idaho. BetterBricks Website The website information, including technical articles, tools and guidelines , access to services, and the calendar, is available to Avista customers. The website is targeted to architects and engineers, building owners; developers and operators. The website can be accessed at www.betterbricks.com. BetterBricks and Lighting Design Lab (LDL) Professional Training and Education BetterBricks and LDL Professional Trainings offer workshops and brown bag presentations to NW architects, engineers, developers , facility managers and other building professionals. These opportunities highlight the benefits of sustainable, high performance buildings and offer tools to help those who build and manage commercial spaces to put strategies for creating high performance buildings into practice. Topics presented by industry experts range from early-stage integrated design strategies to specific design and operations issues such as daylighting or commissioning. These trainings have been coordinated in close collaboration with Avista staff. Several trainings have occurred in Spokane with some participants from Northern Idaho: Moscow Anderson & Wood Construction Meridian 2001/04/10 Li htin Audit Retrofits & Calculations 2001/04/10 Li htin Audit Retrofits & Calculations 2001/04/10 Controls & Commissionin 2001/04/10 Controls & Commissionin 2001/04/10 Controls & Commissionin 2001/04/10 Controls & Commissionin David Ross Eric FI nn Coeur d'Alene School District Coeur d'Alene School District Coeur C'Alene Coeur C'AleneRand Franssen B an Martin Coeur d'Alene School District U otlD Coeur C'Alene MoscowRichard Na Kori Arthur Saint Maries Coeur C'Alene Hu hes Home Center William Dinneen Richard Na Coeur C'Alene MoscowU otlD Idaho Lights Idaho Li hts Blankenshi and Associates Coeur C'Alene Coeur C'Alene Coeur C'Alene Coeur C'Alene Coeur C'AlenePaul Schwartz 2003/10/23 Human Interaction with Light 2003/10/23 Human Interaction with Li ht Kevin Jester 2003/12/04 Design Considerations tor Energy-Effective Lighting Kevin Jester Architects West Int Architects West Int 83844 83642 83814 83814 83814 83844 83861 83815 83815 83844 83815 83815 83815 83814 83814 Longwell Architects BetterBricks Services: Advisors Day Lighting Labs Lighting Design Lab (LDL) BetterBricks services help connect building professionals with the information , tools training and consultation needed to design and construct high performance buildings. The Advisors, Day Lighting Labs and the Lighting Design Lab provide technical assistance to architects and developers. In Northern Idaho, these efforts include: St. Gertrude s Convent: The BetterBricks advisor has been working with Mahlum Architects , out of Seattle , on the renovation and addition to a historic convent in Cottonwood , 10. Upgrade residential annex and build a new conference and retreat center. Client is very interested in energy efficiency and other sustainability issues. Coal-fired boiler serves existing portions of project. University of Idaho: In 2002 the BetterBricks advisor has been working with the University of Idaho of the lighting system for their pool. Building Operator Certification (BOC) Building Operator Certification is a professional development program that teaches facility managers, building operators, maintenance personal and others who monitor commercial building controls how to reduce energy and resource consumption in the facilities they operate. The Alliance provided support through 2002 for NWBOA to market BOC in Idaho. In addition , there have been several classes that have been held in Spokane of the lastthree with some attendees coming from Idaho. In addition, through the Alliance contract with the Association of Idaho Cities, the local government representative has provided ongoing marketing support to encourage local governments to send their operators to BOC training classes. Activity in Idaho includes: Six people attended BOC trainings in Spokane. Attached is a spreadsheet from NEEClisting attendees from Idaho. I checked the names against the utility ZI P code list, and it's not clear that these folks are from Avista. The primary utility listed is Clearwater Power for the students from Orofino and Genesee. I assume that the ones from Coeur Alene are from Kootenai Electric. The one from Nampa is Idaho Power. NWBOA toldme that they tried to hold a BOC training in Coeur d'Alene, but got no marketing support whatsoever from Avista and had to cancel the class. Last First Enroll Date name name Company! Employer Work city State . Work zip Location 1999/09/10 Chatfield John State Hospital North Orofino 83544 Spokane 99 1999/09/07 Dahmen Todd Genesee SD 282 Genesee 83832 Spokane 99 1997/09/18 Martin Bryan Coeur d'Alene SD Coeur 83814 Spokane 02 I #271 Alene 1999/09/10 Waller Ron State Hospital North Orofino 83544 Spokane 99 Small Commercial HV AC O&M Service Pilot This pilot effort is assessing the market opportunities for enhanced O&M services for packaged heating and cooling systems in small commercial buildings. Spokane and northern Idaho is one of the test markets in this pilot effort. The program has been working with Avista staff, Mike Littrel in providing technical and sales training to the Lake City Heating (Coeur D'alene) and Aliant Energy (Spokane). The pilot is targeted 24 customers/units in the Spokane area. Commercial Windows Initiative (CWI) This project works to boost the product availability, consumer demand and market share for energy-efficient manufactured commercial windows that exceed code. CWI has met several times with Avista and has begun working with ,a window manufacturer in Spokane, Marlin Windows who sells windows in northern Idaho as well. Drive Power Initiative Electric Motor Management (EMM) This project works with industrial customers, motor shops and utilities to educate industrial companies on the benefits of planning for motor failure and replacing inefficient electric motors with higher efficiency premium motors or requiring an increased standard for motor rewinds. The program also manages other industrial training efforts, such as Compressed Air Challenge (CAC) classes and Pump System Assessment Tool (PSA T) Trainings. The following direct consultation occurred in Avista s - Idaho territory: Stimpson Lumber - Demonstration and installation of EM2, electric motor inventory software. Consulted with facility personnel on the use of software in making motor repair/replace decisions and generating savings information for upper management for use in developing a motor management policy. Also discussed the potential working relationship with local motor shop, Eastside Electric, and abilities to coordinate inventory information with larger sister plant Atlas. The following seminars and trainings occurred in Avista s territory and were attended by representatives from northern Idaho: EMM Seminar - March 26, 2002 - Spoka , W A: City of Idaho Empire Lumbar . EMM Seminar - May 29 2003 - Spokane, WA: I Strom ElectricATK BacGen This project promotes the use of process controls that optimize electricity use in small to mid-sized wastewater treatment plants. Appropriately adjusted controls can also deliver other benefits by helping plants comply with water quality regulations and better manage sludge accumulation , chlorination and de-chlorination , effluent ammonia and odors. A project in Hayden , Idaho , in Avista s territory, was one of the original BacGen demonstration sites. BacGen completed a system optimization in 2001 , with the folliowing benefits: Average monthly energy costs reduced by 500/0 - from 86,400 kWh to 43 200 kWh. . $2 160/month, or almost $26 OOO/year in savings. Significantly increased treatment capacity for future growth. Additional significant non-energy benefits. Distribution Efficiency Initiative (DEI) The project seeks to identify and support efficiency improvements in utility distribution system design and operation. DEI will demonstrate a variety of voltage regulation strategies to document the cost, benefits and successful practices to achieve efficiency improvements for light commercial and residential customers. Avista Utilities will be one of the utilities participating in the adaptive voltage control pilot. The project recently signed a three-way contract with Avista and PCs Utilidata, the contractor for the pilot with Avista using almost $400,000 of their C&RD allocation to fund their portion of the pilot. 1~~~i~~~~'!~~B~~~I'gN;~~~;!MJ~J' ~:~'~~g~~~~(! ~~;;S!i!!!" Ci i' "?'! Association of Idaho Cities The Alliance funds the Association of Idaho Cities for close to a full-time position dedicated to providing energy efficiency outreach to communities in Idaho. Alliance supported has provided: Support for the Idaho Energy Conference for six consecutive years. AIC representative provides information about Alliance programs to local government officials. AIC gives the Alliance updates on legislative issues. AIC was instrumental in establishing improved energy codes in Idaho. The AIC surveyed communities in northern Idaho to see whether the AIC's outreach efforts on behalf of efficiency led to implementation. So far, they have received three responses: Grangeville updated lighting in city buildings Priest River upgraded their wastewater plant with energy-efficient motors Spirit lake upgraded lighting and heating in the city shop Idaho Code Support The Alliance supports Idaho contractors who conduct code training and educational efforts across the state. One major success from this effort was the recent new code adoption in Idaho. The Idaho legislature passed a bill in March , 2002 to adopt the 2000 International Building Code, which includes the 2000 International Energy Conservation Code (IECC). The IEee covers both residential and non-residential construction and went into affect on December 31 , 2002. The code change will affect about 62010 construction in Idaho. Estimated additional costs to the residential market will be $12. million , with a return-on-investment of $20.5 million in energy savings. On the non- residential side, the additional cost of $713,000 will result in $4.3 million in energy savings. That amounts to a 1 O-year cumulative economic value to the state of $635 million. The residential code revisions address windows and under floor energy efficiencies. Another change will require the use of mastic, a powerful, long-lasting adhesive on ductwork. The code is also updated every three years, so it keeps up with technology changes.