POE chapter finaldoc - Post-occupancy uation (POE) is.doc

Post-Occupancy Evaluation: Issues and Implementation I would like to thank Bill Bordass, Arza Churchman, Joanne Green, Martin Symes and Richard Wener for their thoughtful comments. Some of this material was presented in a different form to the Federal Facilities Council Symposium on Building Performance Assessments: Current and Evolving Practices for Post Occupancy Evaluation Programs, Washington, DC, March 2001.Craig ZimringCollege of Architecture, Georgia Institute of TechnologyRevised May 28, 2001The US is conducting the largest federal construction program since the Second World War, constructing some 160 new courthouses at the cost of $10 billion. After each project is completed, a post-occupancy evaluation team administers surveys, conducts interviews and observes activity in the building. These POEs are used to refine the US Courts Design Guide, the document that has become the key document for judges, architects, project managers and consultants in planning new courthouses.Disney evaluates everything, all the time. The company assesses the experience of customers as they use the parks and other attractions, monitors the relationships between key design decisions and performancesuch as how wide Main Street needs to be to feel comfortably busy during normal attendance levelsand records the performance of thousands of materials and products. The result is that the industrial engineering team can rapidly turn design goals into physical parameters. As a result, they have become key partners with the Imagineering group in planning and designing new properties .A group of UK architectural researchers and building scientists have teamed up with the Building Services Journal to conduct four evaluations annually of innovative sustainable office buildings. The buildings are published in the magazine when they are first opened and then evaluated three years later. Each evaluation involves questionnaires with users and technical assessment of energy use and building performance. The researchers have developed a growing set of benchmarks and the performance of individual buildings are reported with respect to the results for a large sample of similar buildings.Post-occupancy evaluation (POE) grew out of the extraordinary confluence of interests among social scientists, designers and planners in the 1960s and 1970s (see for example,(Friedmann, Zimring, & Zube, 1978; Preiser, Rabinowitz, & White, 1988; Shibley, 1982). Robert Bechtel has estimated that over 50,000 POEs have been completed; a recent web search on Google turned up over 2700 web sites that mention “post-occupancy evaluation” by name. Whereas many POEs are conducted as academic studies, numerous large public agencies have developed POE programs, such as the US General Services Administration, US Courts, US Department of State, US Department of Commerce, US Postal Service, Public Works Canada, California Department of Corrections, State of Minnesota, Government of New Zealand and many others. The California Department of General Services is starting a large POE program and the European Community is initiating a major effort called IANUS, linking building evaluation to the provision of public services focused and on the specification of indicators that show the diversity of interests and perspectives related to different actors in policy-related built environment evaluation. (Symes & Robbins, 2001). The National Research Councils Federal Facilities Council recently conducted a symposium on POE in the federal sector (Stanley & Little, 2001). And, although they do not always call them “post-occupancy evaluations,” many private clients have initiated programs where they systematically assess building performance. POEs are conducted by a wide range of practitioners for many different purposes, and there is no common definition. However, for this chapter I will adapt for POE the definition that Weiss proposed for program evaluation more generally (Weiss, 1997):Post-occupancy evaluation is the systematic assessment of the process of delivering buildings or other designed settings or of the performance of those settings as they are actually used, or both, as compared to a set of implicit or explicit standards, with the intention of improving the process or settings.There are five key aspects of this definition. By “systematic” I mean that the POE follows an explicit, accepted methodology developed for POE or derived from social science, building science, architecture, planning or other discipline. This can be quantitative or qualitative. Whereas POEs have often evaluated buildings, they may also evaluate the details of the process of building delivery, including planning, programming, design, value engineering, construction, facilities management and re-use. Is this the details of the process of building delivery? If so, it might be clearer if it said, which includes, or may include planning POEs assess buildings while they are actually in use, and hence evaluators can assess performance. POE complements other practices such as programming, building modeling, pre-occupancy evaluation and others. Performance standards are not always explicit. They might be implicit and embedded in the methodology used in the evaluation, but they are taken to be objective, or at least intra-subjective, in the sense that they are shared. Most evaluators view performance as multi-dimensional, reflecting the needs or perspectives of a range of stakeholders, such as the organization that occupies the building and the individuals who use or are affected by the building. (The relationship among different performance standards is often a key aspect of POEs, such as understanding and reconciling first-cost with lifecycle cost, or first-cost with user satisfaction.) I am reserving the term “post-occupancy evaluation” for applied studies. Though POE projects might well yield conceptual understanding, and some researchers label their theoretically-driven field studies “POE,” this blurs the focus of POE, which is ultimately aimed at improving the built environment. (However, I do argue that POE needs to be based on theories of building function and theories of organizational learning and change. This is explored further in the Discussion section.)This chapter focuses on the last issue: how POE has contributed to improved buildings and building delivery processes. In the following sections I briefly review the history of POE and its recent development. This is followed by a more detailed discussion of the use of POE. I focus on the distinction between two different kinds of applications of POEs: evaluations that are aimed at supporting a specific project (usually the project being evaluated, though sometimes another project) versus evaluations that are aimed at informing future projects (Zimring, 1981). I emphasize this latter “feedforward” role of POE, which has not received as much discussion as the project support role (Horgen, 1996; Schneekloth & Shibley, 1995). Is this in any way parallel to the difference to a summative vs formative evaluation that Scriven talked about?I provide pay?pay particular attention to the potential, and difficulties faced by large building delivery organizations when using POE for organizational learning. Because the building industry is extremely fragmented among many small clients, design firms, consultants and contractors, the relatively few large public and private organizations can have a disproportionately significant impact on the quality of buildings.Large building delivery organizations have taken the lead in developing POE programs and offer the possibility of having a large major? effect on the built environment. Because the building industry is extremely fragmented among many small clients, design firms, consultants and contractors, the relatively few large public and private organizations can have a disproportionately large significant? impact on the quality of buildings.The following section provides a brief introduction to POE. More detailed discussions can be found elsewhere, such as Friedmann et al., (1978), Grannis, (1994); Horgen, (1996); Kincaid, (1994); Parshall & Pea, (1983); Preiser, (1988); Preiser & Schramm, (1997); Shibley, (1982).History, terms, goals and methodsBackgroundPOE initially developed quickly as a result of the growth of environment and behavior researchsocial scientists, designers and planners who were interested in understanding the experience of building users and in representing the “non-paying” client (Zeisel, 1975). Early POEs were primarily conducted by academicians focusing on the settings that were accessible to them, such as housing, college dorms and residential institutions (Preiser, 1994). During the 1980s, many large public agencies established more structured processes to organize information and decisions in their building delivery processes. As practices such as facilities programming became regularized and were accepted as routine, agencies such as Public Works Canada and the US Postal Service added building evaluation as a further step in gathering and managing information (Kantrowitz & Farbstein, 1996). This development of POE occurred while program evaluation was also rapidly growing. Campbell and many others had been arguing at least since the 1960s that public programs could be treated as social experiments and that rational, technical means could contribute to, or even replace, messier political decision-making (Campbell, 1999). A similar argument was applied to POE, where statements of expected performance embedded in architectural programs could be viewed as hypotheses that POE could test (Preiser et al., 1988). TerminologyThe term “post-occupancy” evaluation was intended to reflect that assessment takes place after the client hads taken occupancy of a building;. tThis was in direct contrast to some design competitions, where completed buildings were disqualified from consideration, or other kinds of assessment such as “value engineering” that reviewed plans before construction. Over the years many theorists and practitioners have grown uncomfortable with the term “POE;”. tThe literal meaning of the term seems to suggest that it occurs after people leave the building and it seems to emphasize evaluation done at a single point in the process. Friedmann, Zimring and Zube (1978) proposed the term “environmental design evaluation.” Other researchers and practitioners have suggested terms such as “environmental audits” or “building-in-use assessment” (Vischer, 1996). More recently, “building evaluation” and “building performance evaluation” have been proposed (Baird, Gray, Isaacs, Kernohan, & McIndoe, 1996). Despite the diversity of the practice, the term “post-occupancy evaluation” remains common for historical reasons and I use it in this chapter for clarityThe Scope of POESome researchers have argued that POE is only one component of an information-based and negotiation-based approach to design decision-making. For example, Bechtel has emphasized the importance of “pre-occupancy evaluation” (Bechtel, 2000). Other authors have suggested that POE cannot be meaningfully discussed as a stand-alone practice but rather need to be considered as one aspect of approaches that include “placemaking” (Schneekloth & Shibley, 1995) and “process architecture” (Horgen, Joroff, Porter, & Schon, 1999). (These are discussed later.) There have been several successful examples where POE has been incorporated into a broader program of user-based programming, discussion and design guide development (Shibley, 1982). For example, the US Army Corps of Engineers initiated an ambitious program of programming and evaluation that resulted in some 19 design guides for facilities ranging from drama and music centers to barracks and military police stations (Schneekloth & Shibley, 1995; Shibley, 1982, 1985). More recently, POE has been seen as part of a spectrum of practices aimed at understanding design criteria, predicting the effectiveness of emerging designs, reviewing completed designs, and supporting building activation and facilities management (Preiser & Schramm, 1997)POE Methods As POE methods have become more sophisticated, several directions have emerged: 1) methods have become more diverse; 2) standard methods packages have been developed.Greater diversity of methodsWith a few notable exceptions, many early POEs primarily focused on assessing user satisfaction, user assessment of building comfort and functionality and on user behavior using self-report methods such as questionnaires and interviews and direct observation of user behavior. More recent POEs now also assess the technical performance of building systems, cost and other factors (Bordass, 1996; Cohen, Bordass, & Leaman, 1996; Leaman, 1995; Raw, 1990). Recently, Judy Heerwagen has suggested that POEs should employ a “balanced scorecard” approach that considers issues such as financial performance, impact of the building on the business process, growth and satisfaction of employees and impact on other stakeholders (Heerwagen, 2001). The balanced scorecard approach is discussed in more depth below.Developing standardized methods Whereas most POEs assess the comfort and satisfaction of everyday building users, as POE methods have developed, more standardized and specialized evaluation approaches have been developed for specific building types such as schools (Ornstein, 1997), healthcare facilities (Carpman & Grant, 1993), environments for young children (Moore, 2000), retail settings (Foxall, 1994; Underhill, 1999), housing (Anderson & Weidemann, 1997) and jails and prisons (Wener, 1993; Zimring, Munyon, & Ard, 1988). Not surprisingly, evaluation of the white-collar work environment has been one of the most active areas for evaluation (Brill, Margulis, Konar, Buffalo Organization for Social and Technological Innovation Inc., & Westinghouse Furniture Systems., 1984; Cooper, 1992; Duffy, 1998; Francis, 1986; Ornstein, 1999; Raw, 1990; Spreckelmeyer, 1993; Stokols, 1988; Wineman, 1986). For example, Vischer created a standardized building-in-use survey that assesses self-reported satisfaction, comfort and productivity (Vischer, 1996). Some researchers have also developed standardized methods for assessing technical performance of buildings such as thermal and energy performance. The “Post-occupancy Review of Buildings and their Engineering” (PROBE) studies conducted by the PROBE team in the UK employ standard questionnaire and technical assessment techniques (Bordass, 1995; Cohen, 1996; Leaman, 1995). The PROBE team has conducted some 18 evaluations of buildings that were published in the Building Services Journal as representing technically innovative buildings. Approximately three years after commissioning, the team returns to the buildings and administers a standard “Building-in-use” user questionnaire and monitors the performance of the heating, ventilating and air conditioning system, records energy use and conducts pressure tests and other measures. The evaluation criteria are part of the standard methods package and allow the team to make links between building design and outcomes. The PROBE team has identified several variables that are good predictors of satisfaction and self-reported productivity. For instance, they have found that issues such as floor plans that offer more access to windows and higher levels of personal control over lighting, heating and noise are strong predictors of self-reported satisfaction. Other standardized evaluation methods focus on more specific aspects of technical performance, such as the Revised Office Environment Survey questionnaire (ROES), which assesses occupants response to indoor air quality (Raw, 1995; Raw, In press). The ROES survey focuses on occupants reports of health, comfort and productivity. The scale has been used many times and has established norms. POE ModelsPOE methodologists have proposed several methodological and conceptual models of POE. For example several authors have emphasized the importance of articulating different levels of POEs with different amounts of activity and resource requirements (Friedmann et al., 1978; Preiser et al., 1988). In his influential book, Preiser advocated three levels of POEs: brief indicative studies; more detailed investigative POEs; and, diagnostic studies aimed at correlating environmental measures with subjective user responses (Preiser et al., 1988). Although there have not been many “theories” of POE POE, many authors have used broader conceptual frameworks to organize their work. Some years ago, Friedmann, Zimring and Zube (1978) suggested that POEs adopt an open systems framework that identifies a “focal problem” and “larger system” based on considering relationships among five elements of building delivery processes: building; users; design process; proximate-environmental context; and social-historical context. Preiser suggested that POE evaluation criteria be based on a habitability framework and has recently expanded this to include a focus on building performance evaluation and universal design (Preiser, 1994; Preiser & Schramm, 1997). As was mentioned above, Heerwagen has recently proposed that Kaplan and Nortons balanced scorecard approach (Kaplan & Norton, 1996) can be used in POE (Heerwagen, 2001). The balanced scorecard approach is a multi-step process where organizational vision and strategy are translated into goals and objectives with quantifiable targets. The balanced scorecard is “balanced” in that it includes both financial and non-financial outcomes and focuses on both routine processes and “breakthrough” performance. Heerwagen has suggested that a balanced POE scorecard can focus on several outcomes: financial, business process, interna