A Methodology for assessing Agile Software Development Approaches

1 A M ETHODOLOGY F OR A SSESSING A GILE S OFTWARE D EVELOPMENT A PPRO ACHES Shvetha Soundararajan Research proposal submitted to the faculty of the Virginia Polytechnic Institute and State University in partial fulfillment of the requirements for the degree of Doctor of Philosophy in Computer Science and Applications Dr. James D. Arthur (Chair) Dr. Osman Balci Dr. Steven D. Sheetz Dr. Todd Stevens Dr. Eli Tilevich March 17, 2011 Blacksburg, VA Keywords: Agile Assessment; Assessing Adequacy, Capability, and Effectiveness; Linkages between Objectives, Principles, and Practices; Indicators ii A M ETHODOLOGY F OR A SSESSING A GILE S OFTWARE D EVELOPMENT A PPRO ACHES Shvetha Soundararajan ABSTRACT Agile method s pro vi de an organization or a team the flexibili ty to adopt a selected subset of principl es and practices based on their cult ure, their values, and t he types of systems that they develop. More specifically, every organization or team implem ents a customized a g ile method, ta ilored to better accommodate its needs. However, the extent to which a customized m ethod supports the organizational objectives, or r ather the ‘goodness’ of that method is quest ionable. Existing agile assessment approaches focus on a comparat iv e analysis, or a re limited in sc ope and ap plication . In this research, w e p ropose a structured, system atic and comprehensive approach to a ssess the ‘goodne ss’ of agile methods. We examine an agile method based on (1) its adequacy , (2 ) the capability of the organization to support the adopted principl es and practice s specified by the method, and (3) the method’s effectiveness . W e propos e the O bjectives, P rinciples and P ractices (OP P) Fram ework to guide our as sessment. The Framew ork identifies (1) objective s of the agile philosophy, (2) principles that s upport the ob jectives, (3) practices that are reflective of the principles, (4) the lin kages between the objectives, principles and practices, and (5) indicators for each practice to assess the effectiveness of the practice and the extent to which the organization suppor ts its i mplementation. In t his document, we discus s our solution approach, preli minary results, and future work . iii Table of Content s 1. Introduction ................................................................................................................................. 1 1.1 Motivation ........................................................................................................................................... 1 1.2 Problem Stat ement .............................................................................................................................. 3 1.3 Issues ................................................................................................................................................... 4 1.4 Solution Approach .............................................................................................................................. 6 1.5 Blueprint ............................................................................................................................................. 7 Refere nces ................................................................................................................................................. 7 2. Background ................................................................................................................................. 9 2.1 Agile Overvi ew ................................................................................................................................... 9 2.2 Agile Assess ment .............................................................................................................................. 13 2.2.1. Agil e Assessment Checklist s .................................................................................................... 13 2.2.2. Agil e Adoption Frameworks .................................................................................................... 14 2.2.3. Agil ity Measurement Approaches ............................................................................................ 16 2.3 Guiding Frameworks ........................................................................................................................ 17 2.3.1. Early Work in Soft ware Measurement .................................................................................... 17 2.3.2. Objectives Princi ples Attribut es (OPA) Framework ............................................................... 19 2.3.3. Evaluation Environment Methodology ................................................................................... 20 2.3.4. Relationship of the OPA Framework and EE Methodology to the OPP Framework ............. 21 Refere nces ............................................................................................................................................... 22 3. Evolving the Assessment Framework ....................................................................................... 24 3.1. The Framework ................................................................................................................................ 24 3.2. Formulated Components of the OPP Fra mewor k ............................................................................ 26 3.3 Proposed Work .................................................................................................................................. 31 Refere nces ............................................................................................................................................... 32 4. Our Proposed Approach to Assessing the ‘Goodness’ of Agile Methods ................................ 33 4.1. Assessi ng Adequacy ........................................................................................................................ 33 4.2. Assessi ng Capability and E ffectiveness ........................................................................................... 35 4.3 Preliminar y Work and Findings ........................................................................................................ 37 4.3.1. FDD Adequacy Assess ment ..................................................................................................... 37 4.3.2 XP Adequ acy Assessment ......................................................................................................... 41 4.3.3 Method A Adequacy Asse ssment .............................................................................................. 44 4.4. Proposed Work ................................................................................................................................ . 49 Refere nces ............................................................................................................................................... 50 Selected Bibliogra phy ............................................................................................................................. 51 5. Substantiating the OPP Framework .......................................................................................... 52 5.1. Substant iating the components of the OPP Framework .................................................................. 52 5.2. Substant iating the assessment process ............................................................................................. 53 6. Conclusion ................................................................................................................................ 55 Appendix A ................................................................................................................................... 56 Appendix B ................................................................................................................................... 58 Appendix C ................................................................................................................................... 62 iv List of Figure s ! Figure 1.1. Core structure of the OPP Framework ......................................................................... 6 Figure 2.1. A generic agile method ............................................................................................... 12 Figure 3.1. The Objectives, Principles and Practices (OPP) Framework ..................................... 25 Figure 3.2. Objectives, Principles, and Practices .......................................................................... 26 Fi gure 3.3. Example linkages in the OPP Framework .................................................................. 27 Figure 3.4 Candidate set of linkages in the OPP Framework ....................................................... 28 v List of Tables ! Table 4.1. Weighted Linkages ...................................................................................................... 34 Table 4.2. Linkage coverage criteria and Likert scale values ....................................................... 35 Table 4.3. Linkage coverage for FDD .......................................................................................... 38 Table 4.4. Adequacy assessment of FDD with respect to its touted bbjectives (using weighted linkage count) ............................................................................................................... 39 Table 4.5. Adequacy Assessment of FDD with respect to its stated principles (using linkage count) ........................................................................................................................... 40 Table 4.6. Linkage coverage for XP ............................................................................................. 42 Table 4.7. Adequacy assessment of XP with respect to its touted objectives (using weighted linkage count) ............................................................................................................... 42 Table 4.8. Adequcy assessment of FDD with respect to its stated principles (using linkage count) ........................................................................................................................... 43 Table 4.9. Linkage coverage for Method A .................................................................................. 46 Table 4.10. Adequacy assessment of Method A with respect to its touted objectives (using weighted linkage count) ............................................................................................. 47 Table 4.11. Adequacy assessment of Method A with respect to its stated principles (using linkage count) ............................................................................................................. 47 Table A.1. Working definitions for the objectives and principles ................................................ 56 Table B.1. Identified linkages between objectives and principles ................................................ 58 Table B.2. Identified linkages between principles and practices .................................................. 58 Table C.1. Observable properties of the people, process, project, and product ............................ 62 Section 1 – Introduction 1 " ! 1. Introduction Agile method s are desi gned fo r cus tomiza tion; th ey off er an organ izat ion o r a te am the flex ibil ity to ado pt a set o f p rinciples and practices based on their culture and values. While that flexibility is consistent with the agile ph ilosophy , it can lead to the adop tion of prin ciples and p ractices tha t can be su b - optimal relative to the desired objec tives. W e q uestion then, ho w can one determi ne if adopted pra ctic es are ‘in sync’ w ith the identified principles, and to what extent those principles support organizational objectives? In this research, we focus on a ssessing the ‘go odness’ of an agile method adopted by an organizati on based on (1) its adequ acy, (2 ) the capa bility of the organiza tion to pro vide th e sup porting environme nt to successfully implement the method, and (3) the me thod’s effectiveness . To guide our assessment , we propose the Objectives, Principles and Practic es (OPP) framewo rk. The design of the OPP framework revolves around the identifi cation of the agile objectives, principles that support the ach ievement of those objectives, and practices that reflect the ‘spirit’ of those principles. Well - defined linkages between the objectives and principles, and between the principles and practices are also established to support the assessment process. We asses s the adequacy of an agile method by traversing the link ages in a top - down fashion. That is, given the set of ob jectives espo used by the agile metho d, we foll ow the linkag es downward to ensure that the appropr iat e princip les are enunc iate d, and that the p roper prac tices are ex pressed. W e asse ss the cap ability of an organization to implement its adopted method and the effectiveness of that implementation by using both a top - down and bottom - up traversal of the linkages. The bottom - up assessment, however, is predicated on the identification of people, process, project and product properties associated w ith each practice that attest to the presenc e and implementation of that practice. We refer to each practice, property pair as an indicator. By following the linkage s upward from th e indicators, we ca n infer the use of proper p rinciples an d the ach ievemen t of desired objec tives. Th is doc ument outlines our propose d resea rch, pre liminary efforts and future directions. 1.1 Motivation Current ly, the numb er of organ izati ons adopting agile methods is inc reasi ng - some of the reasons being (1) the a bility to ac commod ate chang e th roughou t the developm ent lifecycle, (2) improv ed quality, (3) greater return on investment, (4) shorter development periods, (5) improved customer satisfaction, (6) better team morale, (7) reduced waste and (8) better predictability [1- 3] . Agile adoption in an organization is guided primarily by its culture, values and the types of systems being developed by the Section 1 – Introduction 2 # ! organization. More specificall y, when an organiz ation decides to adopt an agile method, we ask the following questions: 1. Does the agil e method adopte d have the potenti al to sati sfy the values (vari ous goals ) of the organization? M ore specifically, does the method have the principles and practices in pla ce to achieve the touted values? 2. Does t he cul ture of t he organi zati on all ow the adop tion and ap plic atio n of t he agile method? Does the o rganization ’s en vironme nt ha ve th e ab ility to supp ort th e im plementa tion of the m ethod? For example, if the people of an organization are resistant to change, getting them to adopt agile metho ds i s a di ffic ult und erta king tha t ma y be fut ile. 3. Is the agile method suited to han dle the types of systems (sm all - , medium - , and la rger - scale, missi on and life - crit ical, etc.) that a re bein g develope d by the organization? For exam ple, can th e metho d suppo rt the devel opment of a lar ger sc ale sy stem th at woul d be built ov er the perio d of three years ? An o rgani zati on’s object ives ar e r efle ctiv e o f i ts cul ture and va lues. However, how c a n one determine if the adopte d agile me thod is con sistent with its objectives ? The agile philos ophy provides an organizat ion or a team the flexi bilit y to adopt a selected subset of principles and practic es. However, more often than not, these customized approaches do not reflect the agile principles associated with the practices. Also, organizations lack the supporting environment to effectively implement the adopted methods. As a result, the benefits afforded by agile m ethods are not fully rea lized [ 4] . H ence, o nce aga in, we qu estion th e extent to w hich a n agile m ethod o r a custom ized approach satisfies the needs of an organization. I n effect, we question the ‘goodness’ of that approach. T he agile philosophy places utmost importance on “working software” being the primary measure of progress. Hence, most ass essment approaches for agil e methods focus on assessing the working software and process artifacts. In particular , they place emphasis on the product and predominantly ignore potential measu res refle ctin g p roce ss, proje ct, and peopl e charact eris tic s. Nonet hele ss, there are select ed approaches that assess the process . The re exist Ag ile Proce ss Impro veme nt Frame works s uch as the Sidky Agil e M easurement Index (SAM I) [ 5 , 6] and Agile Adoption and Improvement Model (AAIM) [7 ] that guide a n organization’s agile adoption and im provem ent efforts. B oth frameworks d escribe levels of agility modeled o n similar concepts found i n the Software Capability Maturi ty Model (SW CMM ) [8] and Capabil ity Maturit y M odel Integra tion (CMMI) [9 ] . Th at is, a set of practices is to b e ado pted b y an Section 1 – Introduction 3 $ ! organization at each level in order to be ‘agile’ at that level. The primary disadvantage of these frameworks is that a set of practices is ‘forced’ on an org anization at defined levels, which compromises the flex ibility o ffered by a gile m ethods. Third party agility measu rement tools s uch a s Com parative Agili ty [ 10 , 11 ] and the Thoughtworks Agile Assessment survey [ 12 , 13 ] focus o n assessing the extent to which an organiz atio n or a team is successfu l in adopt ing and using agil e metho ds. These tools help deter min e the prese nce or absence of pra ctices in an org aniz atio n rathe r than the degree to whi ch those practices are used. Hence, there is a need for a m ore com prehensive approach to assessing the ‘goodness’ of agile methods . 1.2 Problem Statement An agil e me thod is c ompose d of a set of objectives that the m ethod propose s to achieve, princip les that support those objectives, an d prac tices that are reflective of the stated principles. A n o rganization’s culture (people and their attitudes), its values (beliefs a nd ideas about what kind of goals should be pursued), and the types of systems being developed (small - , medium - or larger - scale, m ission and life - critical, network - centric, etc.), define the composition of an adopted agile method. Given the various factors that shape a n agile method within an organiz ation, ho w can we d etermine if that method is ‘in - sync’ with the organizational objectives? In short, how can we assess the ‘goodness’ of an agile metho d? In assessing the ‘good ness’ of an agile me thod, we exam ine the m ethod from three per spectives: 1. We r evie w the compo sit ion of th e ag ile me thod in depe ndent of a ny o rganiza tio nal object ive s. The agile community has endorsed many standalone agile methods such as eXtreme Programming, Feature Driven Development, Crys tal, Lean, etc. H ow c an orga nizations intending to adopt any of tho se ac cepted method s, or attemp ting to desig n a custom ized approach, ensure that the method is sufficient with r espect to meeting its stated objectives? 2. We s tudy an in sta nce of the agi le me thod defined by the o rganizatio n. An in stanc e of an adopted agile metho d sho uld reflec t t he cul ture an d val ues of the organization. H ence, the objectives, principles, and practices touted by the instance m ay differ fr om the composition of the method in its original form. Does the insta nce state desired objecti ves that reflect t he organi zational ob ject ives, an d the principles that support those objecti ves? Section 1 – Introduction 4 % ! Given an instanc e of the agil e method, does the organ izat ion have the abili ty to support its implem ent ation? 3. We o bser ve the effect of t he inst antiat ed meth od w ith in the organiz ation. Did t he appli cation o f t he met hod yiel d th e exp ected / i ntend ed re sult s? Does the organiza tion ’s suppor ting environ ment and/o r the adopt ed pract ices impede or support producing the e xpected re sults? The problem then is to provide organization s with a systematic approach to assessing their adopted agile metho d so tha t they can det ermin e i f t heir met hod i s ‘in - sync’ with their organizational objectives. 1.3 Issues In defining a n approach to assessing the ‘goodness’ of agile m ethods, we have identified the following issues to be addressed : 1. Deter minin g the assessme nt pe rspec tive s A primar y concer n in this resear ch has been to ascer tain the perspect ives from which the ‘goodness’ o f an agile method can be assessed. As discussed previously, we propose to examine (1) a m ethod independent of an organization, (2) an instance of a m ethod within an organization, and (3) the implementation of that instantiated method. T his respectively en tails (1 ) as sessing the sufficien cy of th e m ethod with resp ect to m eeting stated objectives, (2) evaluat ing the ability of an organiza tion t o support the implementati on of the metho d, and (3) d eterminin g if the me thod prod uced the inte nded resu lts. 2. Recogn izing objectives, p rinciples an d practices We recogn ize the e xiste nce o f objec tiv es emb odied by ag ile method s, prin ciple s use d to support the o bjectives, and practices employed that are reflective of the principles. The agile manifesto [ 14 ] provi des four focal values that form the core of the agile philosophy and a set of twelve supporting principles. Ou r task is to derive a set of objectives that are reflective of the a gile philosophy and t he focal values as stated by the manifesto. Also, from the m anifesto, we have to iden tify a compreh ensive set of principles that supp ort the objectives. Section 1 – Introduction 5 & ! 3. Establi shing t he relations hips bet w een t he objec tives, princ iples and pract ices This research focuse s on determining if an agile m ethod adopted by an organization is consistent with the organizational objectives. Hence, given an agile method with its set of objectives, principles and practices, relationships am ong them have to be established in order to determine if there are principles that support the objectives and practices that reflect the princ iples. The re lationships betw een objectives , principles and practices are not explicitly stated in the existing literature. H ence, we have to definitively id entify and establish the rel ationships and provide evidence for the same. We rea liz e tha t for an obje ctive and the set of prin cip les ass oci ate d wit h tha t obj ect ive , one or more principles may support that objective to a greater extent than the others. Simi larly, for eac h prin ciple and the se t of practic es th at are relate d to that princip le, some practices m ay be nece ssary for the achievement of that principle. Recognizing that some relationships between the objectives, principles and practices may be more i mporta nt than the others , we hav e t o ad dress this dispari ty in o ur s olution app roach . 4. Identifying the observ able characteristics of the p eople, process, project an d product Defin iti ve relati onships betwee n observabl e charact eristic s of the peopl e, proces s, project, and product, to practices employed by an organization have to be identified. When use d, each pract ice induc es an assoc iat ed set of obs erv abl e char act erist ics. We need to ident ify these c haracteristi cs so that we can assess: ! An or ganiz atio n’s a bility to support t he implementation of an agile method. - This depends on t he peopl e, proc ess and projec t char acteris tics. ! The exte nt to w hich t he method produce s the expected results. - This depends on t he proce ss art ifacts and pr oduct c haracter istic s 5. Validat ing the assess ment appr oach Valid atin g the a ssess ment ap proach primar ily invol ves (1) sub stant iating t he ass essment perspectives, and (2) gathering evidence for the existence of the objectives, principles, and practices, the relationships among th em, and the ob servable characteristics. To determine the ability of an organiz ation to support an instantia tion of an agile method and evaluate the extent to which that instance produces the ex pected results, we have to study Section 1 – Introduction 6 ' ! the method and its implemen tation within an organizatio n. Hence, to validate the above - menti oned asses sment persp ecti ves, we require the applica tion of the asse ssment approach in an industrial setting. The solution approach described in the next section outlines our proposed approach to add ress the ab ove - menti oned is sues. 1.4 Solution Approach We advoc ate the nee d for a more comp reh ens ive agi le ass ess ment proce ss that asses ses the people, process, proj ect, and product characteristics of organizat ions adopting agile m ethods. In this re search, we describe an approach for ass essing the ‘goodness’ of agil e methods from three perspectives . More specifically, to assess the c ollective ‘goodness’ of a given agile metho d, we address the following three questions: ! How adequate is the method w ith respect to ach ieving its objectives? ! How capable is an organizatio n in prov iding the su pporting environm ent to imp lement its selected method? ! How effecti ve is the im plementa tion of the m ethod in a chieving its objectives? In response to the abo ve quest ion s, we p ropose the O bjectives, P rinciples and P ractices (O PP) framew ork to facilitate the assessm ent of the adequacy, capability and effectiveness of agile m ethods. The framework identifies des irable objectiv es embrac ed by the agile ph ilosophy , an d defin itively links them to principle s that supp ort the achiev ement o f those objectives (Figure 1.1). Similarly, the fr amework also identifies accepte d practices and links those practices to the associated principles. We recognize that some linka ges between the objectives, principles, and practices are more important t han the othe rs. Hence, we assign weights t o the linkage s. The linkag es between the object ives and principles , and between the principles and pr actices, guide t he assessment process. We asses s the adequacy of an agile method by traversing the linkages in a top - down fashion. Given the set of objectives espoused by the agile method , w e Figure 1 .1. Co re structure of the OPP Framewor k Section 1 – Introduction 7 ( ! follow the linkages downw ard to ensure that the appropriate principles are enunciated, and that the proper practice s are expressed. In addition to a top - down examination, the cap ability of an organization to implem ent its ado pted meth od, and the metho d’s effectiveness are assessed using a complementary bottom - up traversal of the linkages. T his begins, however, by iden tifying peo ple, proces s, project an d product characteristics that attest to the use of particular practices. Each practice - property pair is an indicator. Then, by following the linkages upward from the practi ces, we can infer the use of proper principles and the achievem ent of desired objectives. Figure 1.1 shows the core structure of the OPP Framework. The object ives, p rinciple s, practice s, properti es, and the linkages between them are the foundational pieces of the Fram ework (Figure 1.1). We pro pose t o substantiate the components of t he O PP Fram ework by gat hering evidence for their existence from literature and collecting feedback from agile practitioners about the Framew ork itself. W e also plan to identi fy target organizations and assess the adequacy of their customized agile methods and the c apability of those organiz ations to im plemen t the said me thods. A ssessing the effectiv eness of a metho d wou ld i nvolve a lo ngit udin al s tudy, w hich is beyo nd t he scope of our propose d valid ati on go als. 1.5 Bluepri nt The remai nder of this document is or ganized as fol lows: ! In Section 2, w e review some of the existing agile as sessment approaches and th e two approache s guiding the OPP Fr amework. ! Section 3 describe s our pr oposed approac h. It provides an overvie w of the OPP fr amework and its components. ! Section 4 outli nes our ap proach to assessing ‘goodness’ . ! In Section 5, we prese nt our proposed su bstantiation approach . ! Section 6 summarizes our work. References [1] S. Soundararaja n and J. Arthur , "A Soft - Structured Agile Framework for Larger Scale Systems Development," in 16th Annual IEEE Int ernational Conference and Workshops on Engine ering of Comput er - Based Systems (ECBS) , San Francisco, 2009 , pp. 187 - 195 [2] A. Sidky and J. Arthur, "Determi ni ng the Appli cability of Agile Prac tices to Mission and Life - Critical Systems, " presented at the Proce edings of the 31st IEEE Soft ware Engineering Workshop, 2007. [3] C. Schwaber , "The Tr uth About Agile Proce ses - Frank Answers t o Frequently Aske d Questions ," Forrester Research200 7. Section 1 – Introduction 8 ) ! [4] A. Sidky and J. Arthur, "Value - Driven Agi le Adopt ion: Impr oving An Organiza tion's Software Developmen t Approach, " prese nted at the New Trends i n Softwar e Methodol ogies, Tools a nd Techni ques - Proceedings of t he Seventh SoMeT, Sharjah, United Arab Emira tes, 2008. [5] A. Sidky , "A Str uctured Approach to Adopti ng Agile Practi ces: The Agile Adoption Framework, " Ph.D. Disserta tion, Computer Sci ence, Vir ginia Tec h, Blacksb urg, 2007. [6] A. Sidky , et al. , "A disciplin ed approach to a dopting agile practice s: the agile adoption framework.," Innovations in Systems and Software Engineering, vol. 3, pp. 203 - 216, 2007. [7] A. Qumer , et al. , "Agile Adoption and Im provement M odel," in European, Mediterranean & Mid dle Eastern Conference on Inf ormation Systems (EMCIS) , Polytechnic Unive rsity of Valencia, S pain, 2007. [8] M. C. Paul k , et al. , "Capability M aturity Model f or Software, V ersion 1.1," Softw are Engineering Institute1993. [9] W. Roy ce. (200 2) CMM vs. CMMI: From C onvent ional t o Mode rn So ftware Management. Rational Edge . [10] L. Williams , et al. Comparat ive Agil ity . Available: http://comparativeagility. com/ [11] L. Williams , et al. , "Driving Process Impro vement via Co mparative Agility Assess ment," in AGILE Conference, 2010 , 2010, pp. 3 - 10. [12] Thoughtworks. Ag ile Assessment s . Available: http://www.agileassessments.com/ [13] Thoughtworks, " Agile Self Ev aluation Vers ion 1.0 Prepa red for Cipr ian Meste r, Al catel - Lucent," 2 009. [14] Manife sto f or Agile Softwa re Dev elopmen t, 2001. Available: www.agile manifest o.org Section 2 – Background 9 * ! 2. Background The need for a structured, systematic and comprehensive approach to ass essing the ‘goodness’ of agile metho ds forms the basis of our probl em stat ement and solut ion approac h. In this sect ion, we review some of the current agile assessment approaches that have motivated our research. Additionally, to help readers better underst and the work presented in this docum ent, we provide background information about the agile philosophy, its values and principles. W e also present an overview of the Objectives Principles and Attr ibut es Framewor k and the Evaluat ion Environme nt methodolog y t hat guide the construction and design of t he OPP Framework (Section 3) and the ass essment approach (Sect ion 4). 2.1 Agile Overview “Software engineering is (1) the application of a systematic, disciplined, q uantifiable approach to the development, operat ion, and maintenance of so ftware; that is, th e application of engineering to software, and (2) the study of approaches in (1)” [1 ] . The So ftware D evelopm ent Lifec ycle (SDL C) was intr oduced in or der to structu re the softw are developm ent process and ensure that th e pro cess complies with the above definition. The software development p rocess sy stematically beg ins with iden tifying and specifying user requirements a nd then proceeds to the architecting, design, coding , testing a nd maint enance phase s. Traditi onal softw are deve lopme nt metho dolog ies such as the Water fall model [2 ] and the Spiral model [3] have helped reduce the chaotic stat e of software development by focusi ng on: ! Extensiv e planning to outline th e process to be followed, identifying tasks to be comp leted during product development, and det ermining product mil estones, ! Comprehen sive document ation in the form of Software Requirement s Specifi catio n (SRS), high and low - level design do cumen ts, test plan s, etc., ! Gather ing and specifyi ng user requi rements upfro nt befor e proceedi ng to the archi tecting phase, ! Antic ipat ing futu re requir ements and designi ng the archit ecture of the system to accommodate current and future requirements, and ! Process monitoring a nd control, risk m anagement, and software qualit y control and assurance. However , i n th e la st few years, many devel opment te ams have realize d th at these t radi tion al a pproac hes to sof tware development are not suitable to a ll [4 ] . The traditio nal a pproach es a re fo und to be in adequ ate and expensive for development efforts that have to address rapi dly changing require ments. C onventional metho ds att empt to fore see re quir ement s and cre ate the sof twar e syst em archi tectur e upfro nt in or der to Section 2 – Background 10 "+ ! accommodate current and future requir ements. However, when previously unidentified requirements surface, the cur rent arc hitecture may no long er be va lid. The co st of mo difying the architecture, and in turn the code , in or der to accom modate req uiremen ts id entified late du ring the deve lopment life cycle is very high [3] . Creati ng comprehensi ve documentati on is often a wasted activi ty in the face of changing requir ements. Document s have to be maintai ned regul arly t o refle ct any chan ges made to requir ements and des ign. In effect, maintaining co mpr ehensive document ati on i s ex pens ive. Als o, organ izat ions an d te ams invo lved in developing small - scale systems find the extensive planning and documentation efforts cumbersome. The above - mention ed issue s prompte d the need for new cost - effective, lightwe ight me thods to accommodate rapidly changing requirem ents in software systems. This realization motivated the development of the “ Agile ” approach, wh ich is best d escribed by the “Manifesto for Agile Software Develo pment” [5] as given below: “We are uncovering better ways of developing software by doing it and helping others d o it. Through this work we hav e come to value: Individuals and interactions over processes and tools Work ing sof tware o ver comprehensive documentation Customer collabo ratio n over contract negotiation Respond ing t o change over f ollowing a plan That i s, whil e ther e is value i n the items on the right, we va lue th e items on the left more” The core values of t he agil e manife sto are briefly di scussed below: 1. The agile movement focuses on fostering team spirit among the members of the software development team by stressing close relat ionships and open working environments. Human aspects o f the software development process are considered more important than the process itself. 2. The main objective of the software development team is to produce working software at regular intervals. A gile meth ods advoca te the adoption and use of iterativ e and incrementa l development approaches. Wo rking softwa re is used to measure progress and minim al documentation is produced. Section 2 – Background 11 "" ! 3. Relati onships between the customers and the development team are given preferen ce over strict contracts. A gile method s accentuate delivering so ftware that would provide m axim um business value t o the customers thereby reducing the risk of not fulfilling the contract. A lso, the c ustomer is involve d thro ughout the develo pment process, w hich fosters better custo mer - developer rela tionships. 4. The customers and the development team s hould be prepared to m odify plans in order to accommodate change even late in the development lifecycl e. Agile method s are light weight processes that focus on sh ort iterative cycles, direct custom er and user involvem ent, in crementa l dev elopmen t and acco mm odating change even late in the development lifecycle . Also, these methods ar e v alue - drive n; that i s, they foc us on maximizing the business val ue to the stakeholders involved. “ Business value is some thing that delivers profit to the organization paying for th e software in the form of an Increase in Revenue, an Avoidance o f Costs, or an Improvem ent in Service (IRACIS) w hich are discussed below: ! Increase Reven ue – Will this funct ional ity inc rea se th e rev enu e g enera ted by t he syste m? ! Avoid costs – Will thi s f unctional ity improve ef ficiency and r educe wasteful pr ocesses? ! Improve Service – Will th is appli catio n he lp us provi de tim ely servi ce and info rmat ion to othe rs in a better w ay?” [6 ]. In the field of software de velopment, there has been shift in focus from conventional so ftware engineerin g approaches towards agility. The realization that the traditional plan - driven approaches to software engineering are not suitabl e t o all organi zations and teams has motivated the creation of the A gile Mani festo . The creat ion of the manife sto has given ris e to many agil e method s like eXtr eme Programming (XP) [7 -9] , Sc rum [9 , 10 ] , Cry stal methodologies [ 9 , 11 , 12 ] , Featur e D riven Developme nt [9 , 13 ] , L ean D evelopm ent [ 14 , 15 ] , etc. A gener ic agile method In the following parag raphs, we describe a ge neric agile method. T he information pres ented here is based on readings f rom [9 , 16 - 19 ] . Figure 2. 1 shows a gene ric agil e method that fol lows an it erative and increment al approa ch. At th e start of a project, all the stakeholders meet to decide on its scope. The time required to develop the product is est imated and then d ivided into mu ltiple release cycles . D uring each relea se cycle, a potentially sh ippable Section 2 – Background 12 "# ! incremen t of the pro duct is dev eloped. A release c ycle is com posed o f multiple ite rations. A n iteration is a short software development cycle that lasts two to four weeks as shown in Figure 2.1. Durin g the initi al m eeti ng, often refer red to as “ Iteration 0 ”, the stakeholders identify a set of features. These features are sets of functionalit y that deliver business value to the customer and are stored on a “ Project Backlog” (Figu re 2 .1). T hese features are then prioritized and the tim e required for the development of each is determined. A lso, the set of features to be developed during the first release cycle is established . ! Figure 2 .1 A generic agile meth od ! Each feature is decomposed into multiple storie s. Stories are brief descri ptions of user - or customer - valued functionality. The time required to implement each story is determined and the stories are then prioritize d. The p rioritized stories a re stored on the “Iteration Bac klog” (see Figure 2.1), w hich is sp ecific to each iteration . Each story may be fur ther de compos ed into tasks. Tasks are esse ntially ch ecklists for developers, which outl ine the details re quired for implementing t he stories. During an ite ration, the stor ies are developed and are integrated with the existing code base on a d aily basis as indicated by the “Daily Synch” loop in Figure 2.1. Also, as shown in Figure 2.1, the mem bers of the development team meet e veryday for 15 to 20 minutes to discuss: (1) Their accom plishments since the last meeting (2) The issues (if any) that they faced, and (3) Their next tasks to be co mpleted. Agile method s focus on customer communic atio n and collabo rati on. Theref ore, custo m ers or customer representatives are usually present onsite to answer an y question s the developmen t team may h ave and Section 2 – Background 13 "$ ! participate in the product development lifecycle. The customers are responsibl e for creating and recording the acceptan ce criteria for the stories and features. Developers write tests to meet the acceptance criteria defined by the customers. A lso, at the end of each iteration, the customers test the software produced thus far against the acceptance criteria that they outlined previously. Bug s identified w hen testing may be stored in the iteration backlog if they can be fixed during the same iteration o r move d to the project backlog if the y can o nly be remedied dur ing the next ite ration or release cycle. 2.2 Agile Assessment The agile manifes to states that ag ile practition ers valu e “wo rking so ftware o ver com prehen sive documentation” [5] . This value, in con junction with the agile principle that states “o ur highest priority is to s atisfy the customer through early and continuous delivery of valuable software”, primarily guide the choice of agile metrics devised and used [ 20 ] . Henc e, most ass essmen t approac hes foc us on the produ ct . For example, the number of bugs report ed [ 21 ] , the n umbe r of tes ts written for maximum code coverage [ 22 ] , the team veloc ity that in dicates the num ber of story points de livered durin g each iteration [ 20 ] , earned business value [ 11 ] , etc. are so me m etrics use d by agile teams th at focus on the pr oduct. Whil e ess ent ial t o ass ess ing a gil e meth ods, the pr oduct metri cs are b y t hems elves not suff icie nt to provi de a comprehensive assessment. Software En gineering i nvolves people, process, project, and product (the 4 P’s) [ 23 ]. Hence, met rics used in t he as sessme nt of agil e meth ods sh ould i ncorpo rate characteristics of the 4P’s. The OPP Framework that we propose in this research addresses this issue. In this sub - section, we review current agile as sessment appro aches and analyz e their merits and dem erits. 2.2.1. Agil e Assessment Checklist s Agile p ractitioners are aware that in order to continu ously improve an adopted agile m ethod, it is necessary to assess both thei r process and the final product. As a rule , agile teams are asked ‘How agile are you?’ In effect, team s are inclined towards ascertaini ng the extent to which their process is agile. Predominantl y, to assess the agility of their process, they use checklists to determine the presence or absence of practices that are considered ‘agile’. These checklists, however, ignore the principles reflec te d by the practices. Also, most of these checkl ists are tailored to one or more specifi c agil e methods. Some checklists that are commonly used by agile practitioners are (1) the Nokia Test [ 24 ] fo r S crum , (2) How Agile Ar e You (42 - Poi nt Test ) [ 25 ] , (3) the Scrum Master Checklist [ 26 ] , and (4) the Do I t You rself (DIY) Project Process Evaluati on Kit [ 27 ] . T he DI Y Pro ject pro cess Ev aluation Kit is a generic check list that Section 2 – Background 14 "% ! can be adapted for use with any agile process. Additionally, these checklists fai l to assess th e effectiveness of agile methods. 2.2.2. Agil e Adoption Frameworks Wit h t he in creas ing po pular ity of agi le method s, mo re and mo re or ganiz ati ons a re mov ing t owards agil ity. However , rese arche rs have reali zed that many organiza tions and tea ms a dopt and use agile practices witho ut a proper unders tanding of the agile phil osophy . Moreover, many organiza tion s proclaim themselv es to be ‘ agile’ wh en in fact they ar e followin g what is cons idered “A gilefall” (u sing agile terms but essentially following the waterf all model) [ 24 ] . In order to mitigate this issue and gu ide orga nizations in their agile adoption efforts, many agile adoption and process improvem ent frameworks have been developed. In his bo ok titled “B alancing A gility and Discipline – A guide for the perple xed”, Barry Boehm provide s a five step risk - based software development approa ch that is reflective of t he characteristics of both agile and p lan - driven development methodologies [ 28 ] . T his fram ewo rk is help ful for or ganizatio ns that require a hybrid approach com bining the be st aspects of agile a nd plan - driven software development metho ds. However , the primary disa dvant age of this approac h is that the Framewor k provides an overview of the me thod to b e follo wed and not actual practice s [ 29 ] . Organ izations requ ire a tan gible approach to adopting agil e methods. More specifically, they need more guidance about pract ices that are best suited for their requirements before trans itioning to agility. In the next paragraphs , we disc uss two frameworks that provide a structured approach to agile ad option and im provement. Bo th framework s specify practices that an organization should implement in order to embrace agility. Sidky Agile M easurement Index ( SAMI) SAMI was developed at Virginia Tech by D r. Ahmed Sidky as an attempt to guide the agile adoption efforts of an organization [ 29 ] . T he S AM I is based on the idea that th e m ore the n umber of agile practic es adopted by an organization, the greater its agility [ 29 ] . The num ber of agile p ractices ad opted by an organization determines i ts agile potential. Recogniz ing that counting the number of prac tices a dopted is a simplistic measure, Dr. Sidky has created fi ve agile levels. The agile levels defined in SAMI are very similar to the levels of maturity in CMM and CMM I. Each agile level has a set of objectives. These objecti ves reflect the focal values stated in the agile manif esto. Each level is assoc iate d wi th a set of principles stated in the manifesto and practices followed by the agil e com munity [ 29 ] . At each level, a set of pra ctices that a re reflective of the o bjective s defined at that level have been identified. Section 2 – Background 15 "& ! An or ganizati on can choose to a chieve any of the specified levels of agility by adopting all the practices at that level and the levels below . That is, for an organizati on to maintain their agili ty at level 3, all the practices i dentified f or levels 1, 2 and 3 have to be adopted. S AMI outline s a systemat ic and stru ctur ed approac h to guide orga nizat ions in their agi le adopt ion effo rts. The primary di sadvantage of the SAMI is that it compromises t he flexib ility afforded by agile method s. A set of practices for each level is predefined and is “forced” on the organization and thus reducing the flexibility offered by agile methods. M oreover, the established levels may not be reflective of the culture and values of the organizati ons. Agile Adoption and Improv ement Model (AAIM) Qumer’s AAIM [ 30 ] is very simila r to SAM I in that it provid es an agile ado ption and im provem ent framework that specifies varying levels of agility. The inten t of AAIM is to measure the degree o f agility of an agile m ethod. Agility is defined by five parameters, nam ely “flexibility, speed, leanness, r esponsiveness and learning” [ 31 ] . Six levels of agility are defi ned and are fur ther grouped into three agile blocks. Each block and le vel has an associat ed set of objecti ves. A t each block, the degree of agility can be measured quantitatively using the agility m easurem ent mod eling appro ach – the 4 - DAT (4 Dime nsion al An alyt ical Tool ) [ 31 , 32 ] . The 4 - DAT has been designe d to examine agil e methods from four differen t dimensio ns: (1) determi ning the sco pe o f applic ation o f the metho d, (2) agility charac terization based on the fiv e para meters ment ioned pr eviously - flexibility, speed, leanness, responsivene ss an d learning, (3) value - based characterization – identifyin g practices based on the focal v alues stated in the agile m anifesto , and (4) software p rocess characterization – identify ing practice s covering the SDLC phases, project m anagement, configuration management, and process management [ 31 ] . Dimensi ons 1, 3 and 4 invol ve a qualit ative assess ment; di mension 2 i nvolve s a qua ntit ati ve assessmen t. In dim ension 2, the presen ce o r absence of the practices is recorded and the overall score serves as a measu re t o check the exi stence of agil ity in agi le metho ds. AAIM is modele d along similar lines as the SAMI with respect to the CMMI - like agile levels. As menti oned earlier , the disad vant age is reduced flexibility. The degrees of agility are measured by analyzing the adoption of a set of practices. Also, the predefined levels m ay not be ‘in - sync’ with the organizational objectives. A dditionally, the A AIM and SAMI do not provide an indi cation of the effectiveness of the agi le approach under consideration. Section 2 – Background 16 "' ! 2.2.3. Agil ity Measurement Approaches After transit ioning to agilit y, most organiz atio ns are concerned about how ‘good’ has their agil e adoption been - that is, are they achieving what they set out to by adop ting the agile philo sophy? Also, the organizations are intere sted in i dentifying problem areas and issues, and take adequate measures t o solve them. Retrospec tive m eetings at the end of each iteration or releas e cycle help an or ganiz ation or team assess their progress , and ‘fine - tune’ their agile approach . In ad dition to retros pection, teams can em ploy external consultants or tools to help assess their agile process. Most third - party assessment tools/ frameworks are p roprietary and information about them is not readily av ailable. Here, we d iscuss tw o independ ent ag ile asse ssment framew orks available for f ree on the web that ca n be used by a gile practitioner s to assess the agili ty of thei r adopted method. Comparati ve Agilit y Some organi zations focus on being more agile than their com petition rather than striving to be “perfectly agile” [ 33 ] . The Comp arati ve Agility (CA) assessmen t tool (develope d by Kenny Rubin, Mike Cohn, and Dr. Laurie Williams ) is used to help organi zati ons assess their agili ty relati ve to other organizat ions or teams that respo nded to the tool [ 33 ] . C A is a survey - based assessment tool. “Any agil e practi tioner can visit the CA website , answer the survey questions, and receive a free report th at com pares their resu lts to the complete industry dataset” [ 33 ] . T he answers are recorded on a five point Likert scale. Additionally, practitioners could requ est a customized survey. At the highes t leve l, CA identif ies seven dimensi ons that form the basis for the agilit y asses sment. Th ese dimensions can be likened to the Key Process Areas (KPAs) identified by the CMM and CMMI. The seven dimensions are: teamwork, req uirements, planning, technical p ractices, q uality, culture and knowledge creating [ 33 , 34 ] . Each dimen sion is co mpose d of thre e to six ch aracteristic s. Each characteristic has four statements that are assessed by the survey respondents [ 33 ] . “Ea ch sta tement is a practice for w hich the respondent indicates the truth of the statement relative to their team or org anization” [ 33 ] . By utilizing a com bination of dim ensions , charac teristics, a nd state ments, a team or an organi zation, can gauge their agility r elative to that of their competitors or themselves at an earlier time. Through close analysis of the formulated survey questi ons (availabl e from [ 34 ] ), it is evid ent that the statements require an ind ication of p resence or absence of a practice rather than how well is it being used. Responde nts base th eir answer s on their observat ions tha t are not val idated by empirica l data. Als o, the answers to the survey questions are subjective. Moreover, when comparing the agility of two or more organizations, it is unclear if the tool f actors in the difference s in their organizational objectives. Section 2 – Background 17 "( ! Thoughtworks Agile Asse ssment Thoughtworks is a leading agile developmen t and consulting com pany. They have developed an agile assessment survey w hich is available on their website . Agile practitioner s can complete the survey to get a report on the level of agility within th eir organization or team, and also identify opportunities for improvement [ 35 ] . The survey is com posed of tw enty questions covering development and management practices [ 35 , 36 ] . The Thoughtworks Agile Assessment survey questio ns are intended to gather information about the existence and usage of the practices. Th is approach, like CA, does not address assessing the effectiveness of ag ile m ethods in the survey. Howev er, the creators of the tool re cognize and state that the su rvey is intended to o ffer preliminary insights into the agile metho d being follo wed by a team or org anization an d is not a repla cement fo r a custom ized assess ment a pproach. Summary The checklis ts, the agile process improvement frameworks, and the agile assessment tools discuss ed in this sectio n prima rily focu s on th e presen ce or absence of prac tices (bin ary valu es), wh ich is s imilar to the concept o f adequacy that we define in our assess ment appro ach. However, ‘adequ acy’, or any adequa cy - like criteria , is not sufficient to p rovide a comp rehensive assessment o f an a gile me thod. A dequacy is not indicative of the ability of an organiza tion to im plemen t a n a gile method or the effectivenes s of that metho d. Hence , we introd uce two additi onal criter ia or perspect ives for asse ssin g the ‘good ness ’ of agile metho ds, namely capabil ity and effe ctiv enes s. The three perspec tive s are combined to provi de a comprehensive approach fo r assessing the ‘good ness’ of agile metho ds. 2.3 Guiding Frameworks The OPP Framework and the asses sment approa ch develope d in this research are based on the Object ives, Principl es and Attributes (OPA) Framework and the Evaluation Environment (EE) M ethodol ogy. We present these two approaches in this sub - section. W e also p reface this discussion with an overview of some of the earlier software measurement approac hes. 2.3.1. Early Work in Software Measurement Software Metrics have been in use since the mid 19 60’s when the Lines of Code metri c was adop ted as the b asis fo r me asuring program ming prod uctivity and effort [ 37 ] . Th e ratio nale for the d efinition and use of any i ndividual soft ware metric has been either: Section 2 – Background 18 ") ! “(a) the desir e to assess or predict effort/ cost of development processes, or (b) the desire to assess or predic t quali ty of softw are prod ucts” [ 37 ] . Hence, m ost Softwar e M easur ement approac hes adopted by organizations utilize metrics to assess or predict the quality of software products. M etrics to assess quality involve the assessment of software quality objec tives such as maintainabili ty, relia bility, t estabilit y, correctness , adaptabili ty, et c. The primary goal of Software Engineering is to produce a quality product [ 38 ] . T he abs tr act n ature of software has contributed to the difficulties in assessing its q uality. C onsider the following examples discussed in [ 39 ] and [ 40 ] respectively: 1. If th e goal is to assess main tainability, we may conjecture that as the number of uncon ditional branches in a program increases, the more di fficult it would be t o maintain t he program. 2. Schedule slippage adversely impacts the qualit y of the software product. However, one might incorrectly s urmise tha t if a project is o n schedu le, the system being bu ilt is a quality p roduct. In both cases mentioned above, the measures la ck the definitive linkage that directly rel ates the measure to s oftware quality. T hat is, the measu res discussed he re a re margina lly re lated to the goal. Hen ce, predicting the software quality using such measures that are “som ewhat” related to the goal is difficult . Recogni zing the need for defined measures to be dire ctly relate d to software quali ty, researc hers developed m etrics and measurement programs that utilized attributes of the process to assess the product. McCal l’s Fact or/ Crite ria/ Me tric [ 39 ] , Goal Qu estion Met ric (GQM ) [ 41 ] , the Objectiv es Principles and Attr ibut es (OPA) Fr amework [ 40 , 42 , 43 ], and the Ev aluation Environment (EE) Methodology [ 44 - 47 ] are a few assessment approaches that have been developed to address the above - mentio ned issu e. Conside r th e “Goal Question M etric” (GQM) method. “Go al - Questi on - Metri c (GQM) is a par adigm f or the system atic definitio n, establish ment, and exploitatio n of mea suremen t program s supportin g the quantitative evaluation of software proc esses and product s.” [ 48 ] . The GQM method defines a meas urement model cons isting of t hree l evels [ 41 ] : ! Concept ual level (G OA L) - A goal defined for an object (that part of the product or process being observed) based on entities like a) purpose ( the m otivations fo r studying the object) , b) issue (the character istic of the object under study) and c) viewpoint (people interested in st udying the obj ect) Section 2 – Background 19 "* ! ! Operat ional level (QUE STION ) - A s et of que stio ns formulat ed that mus t be an swered to determine if the goals have been reached. ! Quanti tative level (METR IC) - A set of metri cs deri ved from the questi ons to col lect data to answer the same quantitati vely. The coll ected data c ould be objecti ve or subject ive. The GQ M approach to goal - oriented measurement consists of a to p - down decomposition or refi nement of the goals into q uestions an d then into m etrics and a bottom - up anal ysis and int erpretat ion of the collected data [ 49 ] . The data co llected by the u se of metric s a re used to answer the f ormula ted questions, which in turn de termine if the s tated go als have been met. Here, there is an estab lished re lationship between the goal, the questions and the metrics. Hence, we can definitively predict and/or assess quality characteristics using metrics that have been formulated to address specific questions regarding those characteristics. 2.3.2. Objectives Princi ples Attribut es (OPA) Fra mewor k The OPP Framework tha t we propose for the asses sment of the ‘ goodness’ of agile meth ods i s base d on the OPA Fra mewo rk. T he OPA Fr amewo rk ( developed at V irginia Tech) is a structured an d sy stematic approach that serves as a basis for the evaluation o f Software Engineering Method ologies [ 42 ] . The OP A Fram ework identifies: ! Project - level softw are engineerin g objectives like m aintainab ility, re liability, correctnes s, etc. that are commonly recognized by various methodologies ! Principl es th at are reflective of the softw are develo pment p rocess such as informa ti on hiding, structured programming, etc, and ! Attri butes that result from the utilizatio n of the said principles such as reada bility, traceability, etc. [ 40 ]. “Achievement of the objectives comes through the application of p rinciples supported by a methodology. Employment of th e said princ iples res ult in sof tw are p roducts possess ing attrib utes co nsidered desirable and beneficial” [ 42 ] . Also, definitiv e linkages between these comp onents are defined. In ad dition to the above - mentio ned component s, the frame work ident ifie s propert ies that are obser vable charact eris tic s of the p roduct. These pro perties are indicative of the extent to w hich the product possesses the desirable attributes. Each attribute is linked to one or more properties. Each attribute, property pair forms an indicator. Th e indicators fo rm the basis fo r defining m etrics. The v alues obtain ed by the usa ge of these metri cs atte st to the existe nce of desirab le softwar e engi neering attr ibut es in the product [ 42 ] . The se values are propagated and aggregat ed at the princi ples level to present an assessment of the development Section 2 – Background 20 #+ ! process, and the n further aggregat ed at the object ives level to suggest the extent to whic h the objectives are being achieved. A Software Engin eerin g methodolo gy is assessed from two differ ent perspe ctiv es – adequacy and effectiveness. Adequacy denotes the extent to which a methodology can support the achievement of stated project - level goa ls and objectives [ 42 ] . Asse ssing the adequ acy involv es a top - down examination of the linkages b etween the objec tives, princ iples, and attributes to a ttest to the existence o f those F ramew ork components. The evaluation of the adequacy of a methodology is intended to reveal its deficiencies [ 42 ] . Whil e a to p - down traversal of the linkages is adopted for assess ing the adequacy of a m ethodology, the effectiveness of that methodology is determ ined by a top - down and bottom - up examination through the linkages [ 42 ] . “Effectiven ess of a m ethodo logy can be defined as the degree to which that methodo logy produces the desired results” [ 42 ] . Analy zing the m etric values imply the degre e to whic h par ticular properties are obs erved. In turn, this information c an be used to identify a s et of attribut es present. If these attributes are different from the set of attributes identified during the adequacy assessment, it can be implied th at the users h ave failed to adhere to th e principles supporte d by the m ethodolo gy [ 42 ] . 2.3.3. Evaluation Enviro nment Methodology Evaluati on of software systems is becoming increasingl y diffi cult because the systems themselves are large and com plex. Moreover, the proc ess involve s th e measu rement and ev aluation of hu ndreds of qualitative and quantitat ive elements, mandates evaluations by Subject Matter Experts (SMEs), and requires the integration of d isparate measurements and ev aluations [ 44 ] . H ence, “planning and m anagin g such evaluation efforts require a unifying methodolog y and should not be per formed in an ad - hoc manner ” [ 44 ] . In o rder to addre ss the issues in eva luating large an d com plex system s, the Evaluatio n Environment (EE) m ethodol ogy has been developed [ 44 , 46 , 47 ] at V irginia Tech by Dr. O sman Balci. The EE methodology has been designed to conduct evaluations on different type s of projects classified under various domains. In order to accommodate geographically distributed teams, a Web - based client/server software system has also been developed. The Web - based system can be accessed from < https://www.orcacomputer.com/ee/ > . The E E Methodology mandates e valuati ons by SMEs. Some of th e main char acteris tics of the EE Methodol ogy that are adopt ed by the OPP Framewo rk are pres ente d below [ 45 ] : 1. Indicators - “Q ualitative concepts” [ 44 ] such as maintainability, reliability, correctness, etc. cannot be measured directly. The EE methodology proposes to use a hierarchy of indicators to m easure quality attribute s. Each “qualitative concept” that cannot be measured directly is a root indicator [ 44 ]. Hence, to measure each “ quali tative c oncept ”, we decomp ose th e root indicat or into multiple levels o f Section 2 – Background 21 #" ! child indicators such that the i ndicators at the lowest level can be assessed directly. T he indicators at the lowe st level are called the leaf indicators . Branch indicat ors are those that have a parent indicator and at least one ch ild indicator. We as sess only th e leaf indica tors directly. 2. Decompos iti on – As mentioned in the previous paragraph, we deco mpose a root indicat or int o the branch and leaf indicators. The m ultiple levels of indicators form a hierarchy that is an acyclic graph. Also, “ each leaf indicator is m anageable in complexity and is directly assessable by way of testing, direct measurement, analysis, or examination” [ 45 ] . Only the le af indicato rs are me asured a nd evaluated. 3. Integration - When ther e a re multi ple l eaf i ndicat ors for a ro ot indi cat or, the SMEs c rit ical ly weigh the le af indicators by performing a pair - wise compariso n using the Analyt ical Hierarc hy Proces s (AHP). More often than n ot, the leaf indicators are assigned equal weights. Leaf indicators are evaluated by a variety of te chniques such as testi ng, direct measureme nt, analysis, and examination. All the evalu atio ns ar e then i ntegrate d unde r a structur ed framewor k. Leaf ind icat or ev aluat ion scor es are integrated in a bottom - up fashion based on the criticality weightings of the branch indicators. The integration resu lts in an overa ll eva luation score for the root indicator [ 44 , 45 ] . This fina l sco re w ould help us evalua te the root indicator. 2.3.4. Relationship of the OPA Framework and EE Methodology to t he OPP Framework The OPA Framework and the EE Methodology guide the structure of the OPP Framework and the assessment app roach developed in this research. B oth approaches present hierarchical assessment frameworks where qualitative concepts such m aintainability, reliability, correctness, e tc. are evaluated by using defined measures that are directly related to those concepts. We adopt the decomposition strategy touted by b oth frame wo rks to asses s the e xtent to which an agile m ethod achieve s its stated ob jectives. The OPA Framework provides the concept of objecti ves, principles, practices , and linkages and the assessment perspectives of adequacy and effectiveness. Similar to the EE Met hodology, our assessment of capability and effectiveness involve the evaluation of people, proc ess, project, and product properties. We devel op our indi cat or hier archy and the bott om - up assessment approach uti lizing the indicator scores based on the EE Methodology. Section 2 – Background 22 ## ! References [1] IEEE Standards Co llection: Software Engineering, I. S. 61 0.12 - 1990, 1993. [2] W. W. Royce, "Managi ng the developme nt of large soft ware systems : concepts and techni ques, " presente d at the Proceedings of the 9th internat ional conference on Software Engineering, Monterey, Californi a, United States, 1987. [3] B. W. Boehm, "A Spira l Model of Software Development a nd Enhancement ," Compute r, vol. 21, pp. 61 - 72, 1988. [4] R. S. Pr essman, Soft ware Engineering: A Prac titioner's Approach , 6 ed.: M cGraw - Hill I nternati onal Edi tion, 2005. [5] (2001). Manifes to fo r Agi le So ftware Devel opment . Available: www.agilema nifesto .org [6] J. Patton, "Ambiguous Business Value Harms Sof tw are Products," IEE E Softw., vol. 25, pp. 50 - 51, 2008. [7] K. Beck and C. An dres, Extreme Programming Explained: Embra ce Change , 2 ed.: Addison - Wesley Pr ofession al, 2004. [8] R. Jeffr ies. ( 2001). What is Extre me Pro gramming . Available: http://xprogramming.com/xpmag/whatis xp.htm [9] P. Abrahamsson , et al. , Agile Software Devel opment Methods: Review and Analys is . Finland: VTT P ublications, 2002. [10] K. Schwabe r. (200 7). Wha t is Scrum ? Available : http://www.scrumalli ance.org/resources/227 [11] A. Cockbu rn, Crys tal Clear : A Human - Powered Met hodology for Small Teams : Addison - Wesley Prof essi onal, 2004 . [12] A. Cockbu rn, Agil e Software Dev elopme nt : Addison - Wesley, 2002 . [13] S. R. Palmer a nd J. M. Felsin g, A Practi cal Guide to Feature Drive n Development . Upper Saddle River, Ne w Jersey: Prentice Hall PTR, 2002. [14] M. Pop pendi eck a nd T. Poppe ndiec k, Lean Software Developmen t: An Agile Too lkit : Ad dison Wesley, 2003. [15] M. Pop pendi eck. (2003) Lean Sof tware Development . C++ Magazine Met hodology I ssue . [16] G. Smith and A. Sidky, Becoming Agile in an imperf ect world . Greenwich, CT 06830 : Manning Pu blications Co., 2009. [17] D. Leffi ngwell, Scaling Software Agility: Best Practice s for Large Enterprises , 1 ed.: Addison - Wesley Prof essi onal, 2007. [18] J. Highsmith, Agile Soft ware Development Ecosystems : Addison - Wesley, 2002. [19] R. C. Mar tin, Agil e Software Deve lopment: Princ iples, Pat terns, and Pr act ices : Prentice Hall P TR, 2003. [20] D. Nicol ette. (2009). Agile Metrics [Conference Presentation (Agile 2009)]. Available: http://davenicolett e.wikispaces.com/Agile+Metrics [21] R. Petit . (2006). Agile Proce sses: Making Metrics Simple [Internet]. Available: http://www.agilejournal. com/articles/columns/the - agile - man ager/2 4 - agile - processes - making - metr ics - simple [22] A. Ruiz. (2010). Effect ive Code Coverag e (and Metric s in General) . Available: http://alexruiz. developerblogs.com/?p=1421 [23] E. J. Brau de, Software Engineering: An Object - Oriente d Perspec tive : John Wiley & Sons, Inc ., 2001. [24] J. Little. (2007). The Nokia Test Avail able: ht tp://agileconsortium.bl ogspot.com/2007/12/nokia - test.html [25] K. Waters . (2008 ). How Agile Ar e You? (T ake This 42 Point Test) . Available: http://www.agile - software - development.com/2008/01/how - agile - ar e- you - take - this - 42 - point.html [26] M. Ja mes. (2007) . A ScrumMaster’s Check list . Available: http://blogs.danube.com/a - scrummasters - checklist?q=blo g/michaeljames/a_scrummasters_checkli st [27] G. Dinwid die, " http://blog.gdinwiddie.c om/2009/08/18/diy - projectprocess - evaluation - kit/, " vol. 2010, 2009. [28] B. Boe hm and R. Turner, Balancing Agi lity and Discipline: A Guide for the Perplexed : Addison - Wesle y, 2 004. [29] A. Sidky , "A Str uctured Approach to Adopti ng Agile Practi ces: The Agile Adoption Framework, " Ph.D. Disserta tion, Computer Sci ence, Vir ginia Tec h, Blac ksburg, 2007. Section 2 – Background 23 #$ ! [30] A. Qumer , et al. , "Agile Adoption and Im provement M odel," in European, Mediterranean & Mid dle Eastern Conference on Infor mation Syst ems (EMCIS) , Polytechnic University of Va lencia, Spain, 2007 . [31] A. Qumer and B. He nderson - Sel lers, "An evaluation of the degree of agil ity in six agile methods and its applicability for method engineering," Inf. Softw. Techno l., vol. 50, pp. 280 - 295, 2008. [32] A. Qumer and B. He nderson - Sel lers, "A framework to support t he evaluation, adoption and improvem ent of agile methods in pr actice ," J. Syst. Softw., vol. 81, pp. 1899 - 1919, 2008. [33] L. Williams , et al. , "Driving Process Impro vement via Co mparative Agility Assessment," in AGILE Conference, 2010 , 2010, pp. 3 - 10. [34] L. Williams , et al. Avai lable: http://comparativeagil ity.com/ [35] Thoughtworks, " Agile Self Ev aluation Vers ion 1.0 Prepa red for Cipr ian Mester, Alcatel - Lucent, " 2009. [36] Thoughtworks. Avai lable: ht tp://www.agileassessments.com/ [37] N. E. Fe nton and M. Neil , "Soft ware Metri cs: Roa dmap," pr esented at the Proceedi ngs of the Confe rence on The Fut ure of Software Engineering, 2000. [38] S. D. Conte , et al. , Software Enginee ring Metrics and Models : Benjamin/Cmming s Publishing Com pany, Inc., 198 6. [39] J. P. Cavano and J. A. McCall, "A framework for the measurement of software quality", presented at the Proceedings of the software qua lity assurance wo rkshop on Fun ctional and perform ance issues, 19 78 . [40] R. E. Nan ce and J. Arthur, Managin g sof tware quali ty: a measu rement frame work f or as sessme nt and pred ictio n : Springer, 2002. [41] V. R. Ba sili , et al. , "The Goal Question M etric Approac h," in Encyclopedia of Sof tware Engineer ing : John Wiley & Sons, Inc., 1994. [42] J. D. Arthur , et al. , "A Proced ural Approach to Evaluating Sof tware Developm ent Method ologies: The Fou ndation," Virgini a Tech198 6. [43] J. D. Arthur and R. E. Nance, "A Framework for Assessing the Adequacy and Effectiveness of software dev elopment methodol ogies," Vir ginia Tech1991 . [44] O. Balci , e t al. , "Credibility ass essment: a collabo rative evaluation en vironment for c redibility assessm ent of modeling and simulation applications," presented at the Proceedings of the 34th conference on W inter simulation : exploring new frontiers, San Diego, California, 2002. [45] O. Balci . (2008 ). Coll aborative Evaluation Environment : Methodolog y and Web - based Software [CS 6704 Class Notes, Fall 200 8, Depart ment of Computer Science, Virgini a Tech ]. [46] O. Balci and W. F. Ormsby, "Networ k - centric military system architecture assessment methodology," International Journal of System of Systems Engineering, vol. 1, pp. 271 - 292, 2008. [47] M. L. Talb ert, "A Me thodol ogy for t he measure ment and e valuat ion of c om plex system designs," Ph.D. Dissertation, Computer Sci ence, Vir ginia Tec h, Blacksb urg, 1995. [48] A. Fugget ta , et al. , "Applying GQ M in an Industria l Software Facto ry," ACM Transactions on Sof tware Engineeri ng and Methodology, vol. Vol. 7, 1998. [49] F. van Latum , et al. (1998) Adopting GQM - Based Measuremen t in an Industria l Environme nt. IEEE Software . Section 3 – Evolving the Asses s ment Framework 24 !" # 3. Evolving the Assessment Framework Our res earch is moti vated by the la ck of a comp rehens ive approac h to ass essin g agil e method s. We ass ess the colle ctive ‘go odness’ of an agile m ethod adopted by an organiz ation ba sed on (1) its adequacy , (2) th e capability of the organization to provide the supporting environment to implement the m ethod, and (3) the m ethod’s effectiveness . We d efine adequa cy, ca pabilit y and effectiveness as below (definiti ons adapted for current context from [1- 3] : ! Adequacy - Suffici ency of t he method wi th respec t to meet ing stated obj ectives ! Capabil ity – Abili ty o f th e org aniza tion to provide the suppo rti ng env ironme nt c onduci ve t o the implem entation of the m ethod - which is depende nt on its people, proces s and project characteristics. ! Effect iveness – Producing the intended or expected result s - which is dependent on process and product charact eristics We have desi gne d the O bjectives P rinciples and P rac tices ( OPP ) F ramework to assess the ‘g oodness’ of an agile m ethod from the three perspectives d escribed above. This chapter provides an overvie w of the OPP Fr amework and its for mulat ed co mponent s. 3.1. The Framework Figure 3.1 pr ovides an overvi ew of t he O PP framework. The OPP framework identif ies (1) objectives of the ag ile philo sophy, (2) principles that support the o bjectives, (3) practices t hat are reflecti ve of the principles, (4) the linkages between the objectives, principles and practices, and (5) indicato rs fo r each practice to assess the effectiveness of the practice and the extent to w hich the organization supports its implem entation. The cult ure of an organiza tion, i ts values and d esired c haracter istic s of the systems t hat it builds , determine the objectives, principles and practices that it adopts. Our assessment of an agile method is carried out with respect to satisfying those stated objectives. Figure 3.1 illustrates the relationships among the objectiv es, principles , practices and properties. T his relations hip is central to o ur assessm ent proces s and is common to the assessment of adequacy, capabi lity, and effectiveness. To asses s the adequacy of an agile me thod, we traverse the linka ges in a top - down manner from objectives to principles and from principles to practices. The existence of principles and practices supporting the desired objectives is indicative of the adequacy of th e method under considerat ion. Section 3 – Evolving the Asses s ment Framework 25 !$ # !""#$%&&'"(' )$')*+,%' -%./"# 0#%12)34 5)6)7+,+.4 8 ((%3.+9%$%&& :7;%3.+9%& <=+$3+6,%& <=)3.+3%& :7;%3.+9%& <=+$3+6,%& <=)3.+3%& :7;%3.+9%& <=+$3+6,%& <=)3.+3%& <="6%=.+%&' ' "(' <%"6,%>'<= "3%&&' )$#'''<=";%3.' :=* )$+?).+"$),' 52,.2=% @ ),2%'. "'./%' 72&+$%&& @ ),2%'. "'./%' 32&."-%= A 46%'"('&4&. %->' +.&'3/)= )3.%=+&.+ 3& :<<'B= )-%C"=D <="6%=.+%&' ' "('' <="3%&&' )=.+()3.&' )$#'''<="#23.' E$#+3)." =& E$#+3)." =& # Figure 3 .1 The Objectives, Prin ciples and Prac tices (OPP) Fr amework # Addit ional ly, the OPP framewor k ident ifie s people, process, project, and product properties of the practices adopted. These observable characteristics are associated with the practices. Each (practice, property) pair is called an indicator . The se ind icators are e ssential to the assess ment of ca pability and effectiveness. Fi gure 3.1 al so shows the indicators below t he practices for a ssessing capability and effectiveness. People , process and project properties of a practice imply the presence (or absence) of characteristics needed in the supporting environment, i.e., capability. Similarly, other process artifacts and product properties of a practice are representative of the expected results (effectivene ss). Both sets of properties are used to effect a bottom - up assessment of capability and effecti veness by traver sing the li nkages from the a ppropriate prop erties to prac tices, p ractices to principles, and princip les to objec tives. We discuss the components of the OPP framework in the following sub - section. Section 3 – Evolving the Asses s ment Framework 26 !% # 3.2. Formulated Components of the OPP Framework At the heart of the OPP f ramewor k are the object ives, princ iple s, pract ices and the l inkages that tie them together. In dicators ar e identified and are required for the as sessmen t of capab ility and e ffectivenes s. The tasks nece ssary to suff iciently defin e the com ponents o f the OP P framew ork are a s follows: 1 . Derivi ng th e obj ectives and id entifyi ng the supporting principles and the pract ices We recogn ize that each agil e metho d embodi es a set of obj ect ives, whic h are suppo rte d by a se t of underlying principles. Also, there are practices adopted that are reflective of those identified princip les. However , these objecti ves are not explici tly stated. The agile manife sto provides four focal values and twelve p rinciples that d efine the a gile philoso phy. Our work inv olves deriv ing objectiv es reflective of the agile philosophy from the focal values, identifying princip les that su pport the defined objectiv es, a nd identifying practices tha t reflect the pr inciples. As il lustrat ed in Fi gure 3.2, we ha ve i denti fied an i nitial set of o bject ives bas ed on the agil e philos ophy. A supporting aggreg ated set of principles has also been identified from sources including, but not limited to, the agile m anifesto [4] , bo oks [5 -7] , researc h pape rs [8 , 9 ] , expe rience re ports, w hite pap ers [ 10 ] and discussions with indust ry experts. Likewise, we have identi fied a list of practi ces embraced by the agile community. Human – centric V alue – driven Minimal Waste Maximal A dapta bility Continuous innovation and learning Frequent delivery of working software Technical Excellence Simplicity Empowering teams of Motivated Individuals Constant development Pace Accommodating Change Continual stakeholder communication and collaboration Frequent Reflection and Improvement Striving for Customer Satisfaction Individuals & Interactions Working software Customer collaboration Responding to change Iterative and incremental development Evolutionary Requirements Refactoring TDD Automated test builds Pair Programming Minimal BRUF and BDUF JIT Self-Organizing Teams Physical setup reflecting agile philosophy Continuous Delivery Constant Velocity Collocated customers Continuous Feedback Prioritization Daily Progress Tracking meetings Agile Documentation Agile Project Estimation Frequent face- to -face communication Retrospective meetings Client-driven iterations Smaller and frequent releases Appropriate distribution of expertise Software Configuration Management Code ownership (collective and individual) Iteration progress tracking and reporting Adherence to coding standards Agile V alues Objectives Principles Practices # Figure 3.2 . Objectives, Principles, and P ractices Section 3 – Evolving the Asses s ment Framework 27 !& # The obje ctives, principles and pra ctices are t he founda tional pieces used t o assess the a dequacy, capability and effectiveness of an agile method. See A ppendix 3.1 for a list of workin g defin iti ons for th e object ives and principles used in this research. 2. Establishi ng definitive linkages between the identifi ed objectives, principles and practices Linkages between the objectives, pri nciples and p ractice s have to be de fined in ord er to assess the adequacy, capability and effectiveness. These linkages represent defi nitive relationships between the foundational pieces. For example, let us assume that an organization lists maximal adaptabi lity (see Figure 3. 2) as on e of it s objectives . Our workin g defini tion for maximal adaptabi lit y is “flexi bility , abili ty t o accommodat e change and having the freedom to choose practices with respect to the people, process, project, and product” (see Appendi x A). Hence, one underlyi ng princi ple that s upports maximal adaptability is accommodating change. Subsequently, as shown in Figure 3.3, there exists a linkage between the objective ‘Maximal Adapta bili ty’ and the principl e ‘Accommoda ting Change’. We then have a set of practi ces such as evolutionary req uirements [ 11 ], iterative and in cremen tal develo pment [ 12 ] , on - site or co - lo cated customer [ 13 ] , continuous feedback [ 14 ], and minimal Big Requireme nts Up Fron t ( BRUF) [ 11 ] that help realize the principle of ‘Ac comm odating Change’ (also shown i n Figure 3.3). Human – centric V alue – driven Minimal Waste Maximal Adap tability Continuous innovation and learning Frequent d elivery of working so ftware T echnical Excellence Simplicity Empowering teams of Motivated Individuals Constant development Pace Acco mmodating Change Continual stakeh older communication and collaboration Frequent Reflection and Improvement Striving for C ustomer Satisfaction Iterative and incremental development Evolutionary Requirements Refactoring TDD Automated test builds Pair Programming Minimal BRUF and BDUF JIT Self-Organizing Teams Physical setup reflecting agile philosophy Continuous Delivery Constant Velocity Collocated customers Continuous Feedback Product Backlog Daily Progress Tracking meetings Agile Documentation Agile Project Estimation Frequent face-to-face communication Retrospective meetings Client-driven iterations Smaller and frequent releases Appropriate distribution of expertise Software Configuration Management Code ownership (collective and individual Iteration progress tracking and reporting Adherence to coding standards # Figure 3.3 . Example linkages in the O PP Framew ork # Section 3 – Evolving the Asses s ment Framework 28 !' # Following the same systematic process desc ribed pr eviously, we have identif ied a candidate set of linkages b etween all objectives , principles and practice s. The complete set of the identifi ed linkages is shown in Figure 3.4. We have used learning, experience reports, white papers, books, e tc., to identify all and confi rm many o f those rel ationships. Appendix B provides an alternat e repr esentation of the can didate set of identifi ed linkages. Human – centric V alue – driven Minimal Waste Maximal Adap tability Continuous innovation and learning Frequent delivery of working so ftware T echnical Excellence Simplicity Empowering teams of Motivated Individuals Constant development Pace Acco mmodating Change Continual stakeholder communication and collaboration Frequent Reflection and Improvement Striving for Customer Satisfaction Iterative and incremental development Evolutionary Requirements Refactoring TDD Automated test builds Pair Programming Minimal BRUF and BDUF JIT Self-Organizing Teams Physical setup reflecting agile philosophy Continuous Delivery Constant Velocity Collocated customers Continuous Feedback Product Backlog Daily Progress Tracking meetings Agile Documentation Agile Project Estimation Frequent face-to-face communication Retrospective meetings Client-driven iterations Smaller and frequent releases Appropriate distribution of expertise Software Configuration Management Code ownership (collective and individual I teration progress tracking and reporting Adherence to coding standards # Figure 3.4 . Candidate set of linkages in th e OPP Fram ework # Altho ugh not shown in Figu res 3.2, 3.3, and 3.4, the OPP framewo rk suppo rts an additi onal level of linkages betw een every practice and a s et of prop erties th at are ge rmane to the im pleme ntation of the practice are established to assess capabilit y and effectiveness. Those indicators and linkages are discussed next. 3. Identi fying indicator s As dis cusse d previ ously, t o asses s (i) the capabi lity of a n organ izat ion to s upport the im plemen tation of an agile method and (ii) the effectiveness of the m ethod itself, we propose a bottom - up traversal of the established linkages. At the lowest level of the OPP framework we have the practices and hence we begin our assess ment of capabi lit y and effect iveness from the practi ces. However , how do we determine if the organization has the supporting environment to effectively implement a practice and/ or if the practice has produced the intended result s? The answer lies in the i dentificati on of observable properties of the Section 3 – Evolving the Asses s ment Framework 29 !( # practices. These properties are characteri stics of the people, process, project, and product , a nd are specific to each practice. A practice and property pair fo rms an indicator . The first ste p in d efining the indicators is to ide ntify th e ob servable prope rties of the people , proc ess, pr oject, a nd p roduct associate d with each pra ctic e. As di scussed b elow, ind icat ors are requi red to a ssess i. the capab ility of the org anization, an d ii. the effectiv eness of the method Capabil ity We de fine t he cap abi lity of a n orga niz ati on as its abil ity to pro vide t he sup por ti ng e nvi ron ment conduc ive to the im plemen tation of a n agile m ethod. In assessing the capa bility of an organiza tion, we are conc erned with the charact eris tics of i ts internal environ ment . The in ternal e nvironm ent o f an organiz ation is primarily composed of its resources and competencies. More specifically, in an organization, the characteristics of its people, the process (including t he physical envi ronment) that i t adopts, and its projects are reflective of the characteristics of its internal environment. Hence, w e use observable properties of the people, process and project in our assessment of capability. For example, the presence of open physical environments in an organization is in dicative of the organization’s capabilit y to foster face - to - face stakeholde r communication an d collaboration at any given time. We ident ify obser vable pro per tie s of the peopl e, pro ces s and proj ect for every pra cti ce by aski ng the foll owing questions: 1. What spec ial sk ill s or knowl edge do the pe ople in vol ved in the pro jec t nee d to succ ess fully adop t and implement the practice? 2. What cha rac teris tics of t he pr ocess and/ or the env iro nment e xte nd supp ort for the imp lement ation of the pract ice ? 3. Are t here any proj ect specifi c char acteris tics that support or i mpinge on the effe ctiv e rea liz atio n of the pract ice? Conside r the agile practice of pair programming [ 15 ] . Thi s is one of the more popular practice s where programmers develop code in pairs. At any given time, only one programmer can write code. He or she is called the driver. The other programmer, called the observer or navigator , provides assistance to the driver and reviews the code continuously. The programmers switch roles at regular intervals. Asking the questions mentioned above with respect to pair programming provides us with the following characteristics: Section 3 – Evolving the Asses s ment Framework 30 )* # 1. People and their skil l sets: - Will ing nes s of the people to work as pairs and to share their knowledge with their partners. 2. Processes and Physical Envir onment: - Availab ili ty o f onl y one comp uter ter minal for two programmers worki ng as a pair. Effect iveness An agil e method is judge d eff ecti v e if it produces “ working soft ware ” that is a quality product , on time , withi n budge t , and of val ue to the customers . Ass essing the effectiv eness of the agile meth od involves th e identification of properties of process artifacts and the pr oduct, which focu s on the system produced and its value to the stake holders. For example, agile teams strive to maintain constant velocity (the em pirical observation of the work done by a team during an iteration [ 16 ] ). Velocity is reflective of the efficiency and effectiveness of the team . Indicators of constant velocity can be obtained by stud ying Velocity, Burn - Up and Burn - Down cha rts. We ask the foll owing ques tio ns to det erm ine the obser vable pr opert ies of pr ocess art ifa cts and the product: 1. Are ther e any process artif acts generat ed when impl ementi ng the practice ? If so, what are the criteria for the verification and validation of these artifacts? 2. What are t he c riter ia to det ermin e t he s ucc ess ful i mple mentat ion of the p rac ti ce? Wha t pr opert ies are observed in the final product that is a direct result of the successful implemen tation of the pr actice under considerati on? Let us revisit the agile practice of pair programming discuss ed previously . Asking the questio ns menti oned above to determi ne the proper tie s of the product and proce ss artifac ts with resp ect to pair programming provides us w ith the following: 1. Product: - Continuous revi ew of code - Knowledge shar ing among programmers worki ng in pa irs The outp ut is the code it self d eveloped in a modular f ashion. We hav e ident ified an in itia l (al beit , in comp let e) set of obs ervab le pr opert ie s that we present in Appendix C. Section 3 – Evolving the Asses s ment Framework 31 )+ # 4. Defining metri cs for the indicators Once the observa ble propert ies are ident ifie d, our next ste p is to define assessmen t metri cs for each indicator. Th e measure s obtained w ill be indicative of the capability of the orga nization to support the implem entation and the effectivene ss of th e practice s. For example, the num ber o f bugs reported is a metri c associa ted wit h continuo us code revie w. As ment ioned above, cont inuou s code re view i s a produ ct property associated with pai r prog ramming. Henc e, the num ber of bugs reported can be used to assess the effectiveness of pair program ming. If the measure is less than the average number of bugs reported, then we ca n conc lude tha t pa ir progra mming is e ffec tive wit h res pect to conti nuo us code review. Our bott om - up approach to assessing the capabi lity and effectiveness follows the pr ocess outlined by the EE meth odology [ 17 , 18 ] . Give n a practice, we first identify mu ltiple observ able prope rties that attest to its im plemen tation. Then, for each (p ractice, property ) pa ir, i.e. for each ind icator, we evolve a me tric. T o determine the extent to which that practi ce is being used, we aggregate the values of those metric s associated w ith that (practice, property) pair. A fter determining the extent to which the practices of a metho d are eff ecti vely emplo yed, we now have to ascerta in t he ex tent to wh ic h the p rinciples touted b y the ag ile m ethod are in deed reflected by th e prac tices. T his is achiev ed by a further aggregatio n of the degree to which the set of practices a ssociated with each pr inciple are used. Similarly, a f urther bottom - up analysis from the principles to the objectiv es is ca rried ou t in ord er to a ssess the extent to wh ich the stated objectives ar e achieved. 3.3 Proposed Work The OPP Framework provides the foundatio n for our assessment of the ‘goodness’ of agile methods. The objectives, principles, practices, linkages, and indicators are the crucial components of the framework. Now that we have iden tifi ed the component s and establ ished the linkages , we can assess the adequac y of agile methods. How ever, in order to assess the capability a nd effectiveness, the appropriate indicators have to be identified and the m etrics defined. In this research, with respect to the O PP Framework, we propose the f ollowing tasks: ! The set of objectives , pri nciples and prac tices that we have ide ntifi ed is a w orking set. We need t o revisit them perio dically to en sure that the y are still nece ssary and s ufficient. ! Current ly, we have i dentif ied a preliminar y set of li nkages between the ob jecti ves and principle s and principles and p ractices. These linkages definit ively identif y the relationships b etween the foundational pieces of the OPP Framewo rk. Howeve r, we have d efined these linkages through reasoned conjecture. Hence, we n eed to est ablish their comp leteness and validity, and propose to do so by gather ing evidence f rom existing l iterature and confirmation by practitioners in the field. Section 3 – Evolving the Asses s ment Framework 32 )! # ! We de scrib e indi cator s as a (pr actic e, pr oper ty) p air . Our fir st st ep t owards defi nin g th e indi cator s is to ide ntify th e set of observa ble prop erties for each p ractice. T he list o f observable p roperties that w e currently have is inc omplete and we are currently involved in identifyin g the com plete set. ! For each indicator that we define, there is an associated set of metrics to assess the usage of a practice. We still need to define the metrics for eac h indicato r in orde r to facilita te capability and effectiveness assessment. References [1] D. Ulri ch and N. Smallwoo d. (2004) Capita lizing on Capabi lities . Har vard Busi ness Rev iew . [2] J. D. Arthur and R. E. Nance, "A Framework for Assessing the Adequacy and Effectiveness of software development methodol ogies," Vir ginia Tech1991 . [3] J. D. Arthur , et al. , "A Proced ural Approach to Evaluating Sof tware Developm ent Method ologies: The Fou ndation," Virgini a Tech198 6. [4] (2001). Manifes to f or Agi le Software Development . Available: www.agilema nifesto .org [5] P. Abrahamsson , et al. , Agile Software Devel opment Methods: Review and Analys is . Finland: VTT P ublications, 2002. [6] J. Shore and S. Warden , The art of agil e development . Beijing; Sebastopol, CA: O 'Reilly Media , Inc., 2008. [7] G. Smith and A. Sidky, Becoming Agile in an imperf ect world . Greenwich, CT 06830 : Manning Pu blications Co., 2009. [8] A. Qumer , et al. , "Agile Adoption and Im provement Model, " in European, Mediterranean & Midd le Eastern Conference on Infor mation Syst ems (EMCIS) , Polytechnic University of Va lencia, Spain, 2007 . [9] A. Qumer and B. He nderson - Sel lers, "An eval uation of the degree of agi lity in si x agile methods and its app licability for method engineering," Inf. Softw. Techno l., vol. 50, pp. 280 - 295, 2008. [10] C. Schwaber , "The Tr uth About Agile Proce ses - Frank Answers t o Frequently Aske d Questions," For rester Research200 7. [11] L. Cao and B. Ramesh. (2008, January/ Fe bru ary) Agile Requirements Engineering Practices: An Empirical Study. IEEE Software . [12] M. Pop pendi eck a nd T. Poppe ndiec k, Lean Software Developmen t: An Agile Too lkit : Addison Wesley, 2 003. [13] A. Cockbu rn, Agil e Software Dev elopment : Addison - Wesley, 2002 . [14] T. Dingsøyr and G. K. Hanss en, "Extending Agile Methods: Postmortem Revi ews as Extended Feedback," i n Advances in Learning So ftware Organiz ations . vol. 2640 , S. Henninger an d F. Maurer, E ds.: Springer Berlin / Heidelberg, 200 3, pp. 4 - 12. [15] L. Wil liams and R. R. Kessle r, Pair programming illuminate d . Boston: Ad dison - Wesley, 2003. [16] D. Nicol ette. (2009). Agile Metrics [Conference Presentation (Agile 2009)]. Available: http://daven icolette.wikispaces.com/A gile+Metrics [17] O. Balci , e t al. , "Credibility ass essment: a collabo rative evaluation en vironment for c redibility assessm ent of modeling and simulation applications," presented at the Proceedings of the 34th conference on Winter simulation: exploring new frontiers, San Diego, California, 2002. [18] M. L. Talb ert, "A Me thodol ogy for t he measure ment and e valuat ion of c omplex syst em de signs ," Ph .D. D isser tati on, Computer Sci ence, Vir ginia Tec h, Blacksb urg, 1995. Section 4 – Our Propose d Approach to Ass essing t he ‘Goodn ess’ of Agi le Methods 33 !! " 4. Our Proposed App roach to Assessing the ‘Goodness’ of Agile Methods Given the OPP Framework descr ibed in Section 3, we asses s the ‘goodne ss’ of an agil e method by evaluating its adequacy, the capability of the organization to provide the supporting environment to implem ent the method, and the effectiven ess of that method. The Framework provides us with the foundational pieces, n amely the ob jectives, principles, practices, link ages, and indicators needed to guide our approach to assessing ‘goodness’. 4.1. Assessing Adequacy Adequac y is define d by the suffic iency of an agil e method with res pect to meeti ng its stated obje ctiv es. It is indepen dent of an organization. M ore specifical ly, we can assess the adequacy of standalone agile metho ds such as eXtre me Progra mming (XP) [ 1-3 ] , Lea n [4] , Crysta l [ 5] , F eature D riven Develop ment (FDD) [1 , 2 , 6 ] , etc. with resp ect to th e agile values an d principle s e ach espou ses. That is, given an objective of a method, are the necessary principles also present that are prescribed by the Framework? Then, for each pri nciple enunciated by the Framework, are the re commended practi ces touted by the agile metho d? I f neces sary pri nciples and pra ctic es are mi ssin g, t hen ade quacy is suspect . We begi n asses sin g the ade quacy of an agil e method by anal yzi ng and iden tifyi ng its ob jec ti ves , principles, and practices. We determine the linkages between its foundational pieces. The adequacy assessment process is a top - down approach that does not involve indicators. From the linkages in the OPP Framework, we ascer tain the expected number of linkages between ea ch objecti ve and its associated set of principles, and each principl e and the practices that are reflective o f that principle. W e then determine the linkage coverage for each ob jective and principle, which is the ratio between the actual numb er of linkages f ound an d the exp ected num ber of lin kages. W e map the linkag e covera ge to a L ikert scale to determine the ad equacy of the m ethod und er consideration with respect to its objectives and principles. Based on the linkage coverage and the Likert scale value s, we asses s the adequacy of a method with respect to its objectives. Weighted Linkages When for mula ting our a pproach to assessing adequacy, we realized that with respect to the linkages between the objectives and principles, some principles are reflective of an objective to a greater extent than the others. Conside r th e o bjective ‘Max imal Adap tability’. From Tabl e B.2 (see Ap pendix B), we know that ‘Maximal Adaptabi lity’ i s supported by the following pr inciples: Section 4 – Our Propose d Approach to Ass essing t he ‘Goodn ess’ of Agi le Methods 34 !# " ! Frequent Deliv ery of Worki ng Software ! Technical Excellenc e ! Simplicity ! Accommodati ng Change, and ! Frequent Refle ction and L earning Here, ‘Accommodat ing Cha nge’ is more reflective of ‘Maximal Adaptability’ than the others. Hence, it carries more weight than the other principles that are associated with that objective. For our assessment, we assign such linkage s a weig ht tha t is twice that of t he li nkages betw een the other principles associated with t hat obj ective . Here, ‘Accom moda ting Chang e’ is assigned a weight of 2 , and the other princip les a weight of 1 each w ith respect to “Maximal Adaptability”. Table 4.1 below shows the linkages between the objectiv es and principles. The entries i n bold represent linkages that carry a weight twice that of the other linkages . Table 4. 1 . Weighted Linkage s Objectives/P rinciples Freq. Delivery Tech. Excellence Simplicity Emp. Teams of motivated indi viduals Const. Development Pace Acc. Change Continual stkhlder comm. and coll. Freq. refln and learning Striving for customer satisfaction Human - centric X X X X Value - driven X X X X X X Mini mal wast e X X X X Maxim al Adapta bilit y X X X X X Continu ous Innovation and Learning X X These weighted linkages play a major r ole in the a ssessment of t he extent to which t he touted objectives are supported by a method. Similarl y, some practices reflect the ‘spirit ’ of a principle to a gr eater extent than the others. More specifically, there is a set of necessary and sufficient practices that are associated w ith each p rinciple. Current work is underway to assign weight s to the linkages between principles and practi ces. A summar y of th e st eps involved in assessing the adequacy of an agile method is given below: 1. We examin e the agi le meth od under consi der ati on, and ide nti fy its obj ect ive s, pri nci ple s, and practices. 2. From the OPP Framework, we determine the l inkages tha t exist between the obje ct ives, principles and practices stated by the method. Section 4 – Our Propose d Approach to Ass essing t he ‘Goodn ess’ of Agi le Methods 35 !$ " 3. From the OPP Framework, we obtain the expected number of linkages between the identif ied objectives, principles, and practices (weighted for the linkages between the objectives and principles). M ore specifically, if the method under consideration touts ‘Conti nuous Learning and Innovation’ as on e of its objectives, from the O PP Fram ework, we know that num ber of linkages associated with that objective is two and the weighted count is th ree (Table 4.1). H ence, for the above - mentio ned objec tiv e, t he expe cted number o f we ight ed l inka ges is thre e. 4. We est abl ish th e lin kag e cove rag e as a rati o o f the number of link age s pres ent (fr om the method under consideration) and the number of linkages expected (from the O PP Framework). Currently, we use weighted linkag e covera ge only for the linkag es between objectiv es and princi ples. We simply count the number of linkages between the principles and practices. 5. After determi ning the li nkage cov erage for each obj ecti ve a nd principle, we map the values to a Likert scale. Pr esently , we use the cr iteri a given bel ow i n Table 4.2 t o map the percentage value of the linkage coverage to the Likert scale values. The criteria and the Likert scale values used here are subj ective and ar e not yet final. Table 4. 2 . Linkage coverage c riteria and Likert sc ale values Linkage Coverage Criteri a Likert Scale Values 81 - 100 % Strongly 61 – 80% Very we ll 41 – 60% Sufficient ly 21 – 40% Somewhat 0 – 20% Very l ittle C urrently, for each objective and pr inciple, w e provide a Liker t scale value based on the criteria given in Table 4.2. These values are indicative of the extent to which the metho d being as sessed is adequate with respect to each objective and principle that it states. We plan to outline an app roach for represe nting the adequacy assessment results. We have ass essed the adequ acy of thre e agil e metho ds – FDD , XP, and Method A. A detailed account of the assessm ents is pro vided in S ection 4.3. 4.2. Assessing Capability and Effectiveness Unlik e adequac y, we cannot asse ss the cap abil ity and effect ivene ss of a sta ndalo ne agil e method th at is independ ent of an org anization, because both capability and eff ectiveness require an orga nizational Section 4 – Our Propose d Approach to Ass essing t he ‘Goodn ess’ of Agi le Methods 36 !% " perspective. Assessing the capability of an organization to support the im plementation of a method and the effectiv eness of th e metho d itself requ ires both a top - down (objectives " principles " practices) and bottom - up (properties " practices " principles " objectives) traversal of the lin kages. The dif ference between the assessment of capability and effectiveness lies primarily in the type of indicators used - people, process and project indicators for capability, and process artifacts and product indicators for effectiveness. It is required to ensu re th e e xistence of a p ractice before procee ding to assess the e xtent to w hich it is implem ented in an orga nization. A lso, it is necessary to con firm that the practic e is sup ported b y the organization bef ore determining t he degree to whi ch it is being used. Hence, within an organization, we assess the adequacy of the adopted agi le method, followed by t he capability of the organization to implem ent that me thod, and f inally the m ethod’s ef fectiveness . Our bott om - up approach to assessing the ca pability and effectiveness follows the process outl ined by t he EE methodol ogy [7 , 8] . Given a practic e, w e first identify multiple observable properties (people, process, project and product charact eristics) that attest to its implem entation. Then, for each (practice, property) pair, i.e. for each indicator, we evolve a metric. To determine the extent to which that practice is being used, we aggregate the values of those metrics associated with all the practice - propert y pairs specific to that practice . The aggr egated valu es are indica tive of ! the extent to which th e implem entation of the practice is supporte d by the or ganization, and ! the effectiv eness of the practice itself . After determin ing t he extent to which the practi ces of a method are eff ecti vely employed, we as sess the extent to which the principles touted by the agile method are indeed reflected by the practices. This is achieved by a further aggregation of the degree to which the set of practices associated with each principle are used. Simil arly, a further bottom - up anal ysis from the principles to the objectives is carr ied out in order to assess the extent to which the stated object ives are achi eved. We rec ogni ze that th e me trics t o be defi ned for each indic ator would yield va lue s t hat mayb e subj ectiv e, objective, num erical, binary, range values, etc. Hence, we need to map the different types of v alues obtained onto a uniform scale of measurement to pe rform th e aggre gation. H owev er, we have no t yet defined that necessary mapping approach. C urrent research is underway to identify an appropriate uniform scale of measurement. The EE Methodol ogy sugg ests one viable approach. Section 4 – Our Propose d Approach to Ass essing t he ‘Goodn ess’ of Agi le Methods 37 !& " 4.3 Preliminary Work and Findings In this research, we propose to assess agile m ethods from three perspectives – adequacy, capability and effectiveness. The adequacy of an agile m ethod is assessed independent of any organizati ona l objectives. Unlik e adequac y, the capabili ty of an organizat ion to support the implemen tati on of a n agile method and the effe ctiveness of the method itself, a re assess ed from an organizatio nal pers pective. O ur pr eliminary work focuses on adequa cy. More s p ecifically, we have examined the adequacy for t wo standalone metho ds – XP [1 - 3] and FDD [6 ] , a nd fo r Me thod A th at is an instantiatio n of XP in a n org anization . XP is an agile method that best reflects the agile philos ophy. Me thod A is an XP varian t impleme nted in an organization. FDD is int ended for mediu m to l arge r sc ale s ystems de velo pment and h ence tout s a b lend of agile values and conventional software engineering principles. The followi ng sub - sections summ arize the assessm ent results that w e have obtained by applying ou r adequacy assessment process to FDD , XP and M ethod A. For each of the three methods, w e identified their toute d objectiv es, princip les and practices, and esta blished th e linkage s betwe en them . We then determined the linkage coverage and mapped the values to the Likert scale descr ibed in Section 4.1. 4.3.1. FDD Adequacy Assess ment Feature Driven Devel opment ( FDD) is an agil e method devel oped by Jeff De Luca and Peter Coad. It is an iterative and incremental app roach that is more structured than the other ag ile methods and is desig ned for scalability. Hence, it is commo nly used for developing medium to larger scale system s. In the w ords of Peter Coad, “FDD ha s just enough process to ensure scalability and repeat ability and encourage creativity and innovation all along the way” [1] . FDD does not addres s the proc ess of requ irements engineerin g. It fo cuses only on the design and codi ng phases of development and does not span the complete development process. Therefore, other Require ments Engineer ing methods have to be used in conjunc tion with the FDD approach to developing software. Adequacy Assessment As outli ned in Sectio n 4.1, we begin the asses sment approach by analyz ing the method under conside ration. W e identify the objec tives touted by the method , the sup porting prin ciples, and the adop ted practices, and identify the linkages between them. As discussed previously, w e assign weights to the linkages between the obje ctives and principles. For eac h stated objective, we compute the weighted linkage co verage tha t is the ratio betw een the o bserved w eights fo r the linkage s and the ex pected w eights. Section 4 – Our Propose d Approach to Ass essing t he ‘Goodn ess’ of Agi le Methods 38 !' " Observ atio ns ab out FDD The success of FDD, like other agile methods, is largely due to recogniz ing softw are development as a human endeavor. Hence, ‘good’ people become the primary asset to the organization adopting FDD. The creators of FDD are of the opinion that the people are more importan t t han the proc ess. FDD is diametr ically opposite to t he other a gi le metho ds in its prac tice of upfront model ing. The ot her method s, XP f or example , dec lar e that the arch itec ture of a syst em should em erge from an iterative process over se veral increments. FDD on the other hand places emphasis on “getting it right the fir st time” [ 1 , 6] . Hen ce, during the initial model ing phase , a li ghtweigh t a rchi tect ure is cre ated and the cl ass skeleto ns a re writ ten. The m ajor disadvantag e of the upfront modeling and design effort s is that these practice s reduce the ability to accom modate cha nge later in the development cycle. The architecture of the system is ‘frozen’ at the beginning of the development effort, and accommodating major changes would be expensive. Howev er, the fact that FDD supports upfront modeling and design is the prim ary reason for its suit abili ty for developing medium and l arger scale s ystems . The FDD approach to developing software is not “lean” [9 ] . More specifically , it proposes to bui ld more than what is needed. FDD does not accommodate client driven iteratio ns. Proponent s of FDD believe in the development team dri ving the i terations as opposed to the other agile methods where the clients stipulat e the order in which the features are developed. Based on our anal ysis and obse rvati ons, we have iden tifi ed a set of object ives , principles , an d p ractices that we conj ect ure are touted by FD D. Table 4.3 provides a list of objectives and princi ples stated by the metho d. A lso, the l inka ge c overa ge a nd t he weigh ted lin kage coverage values are included in Table 4.3 . Table 4. 3 . Linkage coverage fo r FDD Objectives Linkage Coverage Weighted Coverage Human - centric 3/ 4 4/ 6 Value - driven 5/ 6 7/ 8 Maxim al Adapt abilit y 3/ 5 3/ 6 Continu ous Inno vation and Learning 2/ 2 3/ 3 Section 4 – Our Propose d Approach to Ass essing t he ‘Goodn ess’ of Agi le Methods 39 !( " Principles Linkage Coverage Frequen t Delivery of W orking So ftware 8/ 8 Technical Excelle nce 12/ 18 Empowering t eams of mot ivated i ndividual s 4/ 5 Constant Development Pa ce 7/ 10 Frequent Refl ection and Improvement 6/ 9 Striving for Customer Satisfacti on 3/ 5 Based on the link age co ve r age values provided in Table 4. 3 , we m ap thes e values for FD D to a Lik ert scale as described previ ously in Section 4.1. Tables 4.4 and 4. 5 provide the mappings of the l inkage coverage values to the Likert scale for objectives and principles respectivel y. Table 4. 4 . Adequacy assessm ent of FDD w ith respect to its tou ted bbjectives (using weighted linkag e count) Human - centric X Strongly Very we ll Sufficient ly Somewhat Very l ittle Value - drive n X Strongly Very we ll Sufficient ly Somewhat Very l ittle Maxi mal Adapt abil ity X Strongly Very we ll Sufficient ly Somewhat Very l ittle Continu ous Inn ovatio n and Learnin g X Strongly Very we ll Sufficient ly Somewhat Very l ittle From Table 4.4 , we know that a mong the ob jectives touted by FD D, ‘V alue - driven’ and ‘Continuous Innovation and Learning’ are su pported to a grea ter extent th an the oth ers. As m entioned previou sly, FDD does not focus on ‘being lean’. That is, F DD does not state the objective of ‘Minimal waste’ and its associated principle of ‘Simplici ty’. FDD is intended for medium to larger scale syst ems developme nt and Section 4 – Our Propose d Approach to Ass essing t he ‘Goodn ess’ of Agi le Methods 40 #) " is m ore structured than the other a gile methods. More sp ecifically, its cons truction and de sign minimizes the ability to accomm odate change . Hen ce, FD D s upports ‘Maxim al A daptabilit y’ to a lesser extent (Table 4.4). Table 4. 5 . Adequacy Assess ment of FDD with respect to its sta ted principles (us ing linkage count) Frequent de livery o f working s oftware X Strongly Very we ll Sufficient ly Somewhat Very l ittle Technica l Excel lence X Strongly Very we ll Sufficient ly Somewhat Very l ittle Empowering teams of M otivat ed Indiv iduals X Strongly Very we ll Sufficient ly Somewhat Very l ittle Constant development pace X Strongly Very we ll Sufficient ly Somewhat Very l ittle Frequent Ref lection and Improvement X Strongly Very we ll S ufficiently Somewhat Very litt le Striving f or Customer Satis faction X Strongly Very we ll Sufficient ly Somewhat Very l ittle Table 4.5 provides an overview of the extent to which FDD suppor ts its sta ted princ iple s. FDD is an iterative a nd incre mental software dev elopmen t approa ch that focuses prim arily on ‘Frequ ent De livery of Work ing Softwa re’ an d to uts ne ces sar y prac tic es tha t are r eflec tive of the said principle (Table 4. 5 ). A s menti oned previ ousl y, cust omers are not invo lved t hroug hout t he devel opment lif ecycle and the ite rations are not client - driven. That is, ‘Striving for Customer Satisfacti on’ is supported to a lesser extent as shown Section 4 – Our Propose d Approach to Ass essing t he ‘Goodn ess’ of Agi le Methods 41 #* " in T able 4 .5 . T eams maintain a con stant development pace. At regu lar intervals, the stakeholders meet to reflect and ‘fine - tun e’ their proc ess. Summary As discu ssed previou sly, FDD is more suited for m ediu m to large r scale syste ms. It is more structu red than the other agile metho ds. By de sign, the method does not state certain objectives such as ‘Minimal Wast e’ an d ‘ Maximal Adapt abili ty’. Al so, cert ain c las sic a gile pract ice s suc h a s ‘ Clien t - driven Iterations’, ‘Refactoring’, ‘Min imal Big Requirem ents U p Front and Big Design Up Fro nt’, and ‘T est D riven Develo pment’ are not suppor ted. Hence, some of the touted objec tive s and principl es are achieve d to a lesser extent. Also, from our adeq uacy assessm ent r esults f or F DD, XP and M ethod A, we surmise that FDD is the least adequate with res pect t o its stat ed objectives than the ot her two methods. 4.3.2 XP Adequacy Assess ment eXtreme Programming (XP) is designed for small to medium teams w hich have to develop software quickly in the face of rapidly changing requirements. XP is based on four value s: comm unication, simplicity, feedback and courage [2 , 3 ] and “It works by bringing the who le team together in the presence of simple practices, with enough feedback to enable the team to see where they are and to tu ne the practices to their unique si tuation” [2]. The customers are actively involved in the development proces s. XP came int o exist ence with software development practi ces found eff ecti ve du ring the prev ious decades. Observ atio ns ab out XP The desi gn of the system is usu ally kept sim ple. Develo pers work in pair s and impl ement features using T est Drive n Deve lopment (TDD). The customers write accept ance crit eria agai nst which the sys tem is test ed at the end of each iteration and rel ease cycle. Each it erati on lasts 2 - 4 weeks. XP is a set of pract ices that can be adopted by agile practit ioners who may choos e to use some or all of the practices or may customize the method to suit their needs. XP is d esign ed for small te ams. Howe ver, cert ain XP pract ices can be used in larger scale systems development as well (for examp le, pair programming). Section 4 – Our Propose d Approach to Ass essing t he ‘Goodn ess’ of Agi le Methods 42 #+ " Adequacy Assessment Table 4.6 provides a summary of the linkage coverage and weighted coverage values for the objectives and principles supported by XP. Table 4. 6 . Linkage coverage fo r XP Objectives Linkage Coverage Weighted Coverage Human - centric 4/ 4 6/ 6 Value - driven 6/ 6 8/ 8 Mini mal Waste 4/4 5/5 Maxim al Adapt abilit y 5/ 5 6/ 6 Continu ous Inno vation and Learning 2/ 2 3/ 3 Principles Linkage Coverage Frequent Deli very of Worki ng Software 8/ 8 Technical Excelle nce 15/ 18 Simplicity 6/ 6 Empowering t eams of mot ivated i ndividual s 4/ 5 Constant Development Pa ce 10/ 10 Accommodat ing Change 11/ 11 Continu al sta keholde r communic ation and collaboration 6/7 Frequent Refl ection and Improvement 8/ 9 Striving for Customer Satisfacti on 5/ 5 Mappi ng the li nkages cove r age values for XP from Table 4.6 to th e Likert scale used in this res earch (see Section 4.1), we obtain t he no minal values sho wn in Table s 4. 7 and 4.8 . Table 4. 7 . Adequacy assessm ent of XP with respect to its touted o bjectives (using w eighted linkage co unt) Human - centric X Strongly Very we ll Sufficient ly Somewhat Very l ittle V alue - driven X Strongly Very we ll Sufficient ly Somewhat Very l ittle Section 4 – Our Propose d Approach to Ass essing t he ‘Goodn ess’ of Agi le Methods 43 #! " Mini mal Waste X Strongly Very wel l Sufficiently Somewhat Very li ttle Maxi mal Adapt abil ity X Strongly Very we ll Sufficient ly Somewhat Very l ittle Continu o us Innovation and Learning X Strongly Very we ll Sufficient ly Somewhat Very l ittle XP is an agile method that best reflect s the objecti ves of the agil e philo sophy. Al l of its stat ed objec tive s are maximally supported by associated princi ple s (Table 4.7 ). Table 4. 8 . Adequcy assessm ent of FDD w ith respect to its stated principles (using lin kage count) Frequent de livery o f working s oftware X Strongly Very we ll Sufficient ly Somewhat Very l ittle Te chnical Excellence X Strongly Very we ll Sufficient ly Somewhat Very l ittle Simplicity X Strongly Very we ll Sufficient ly Somewhat Very l ittle Empowering teams of M otivat ed Indiv iduals X Strongly Very we ll Sufficient ly Somewhat Very l ittle Constant development pace X Strongly Very we ll Sufficient ly Somewhat Very l ittle Section 4 – Our Propose d Approach to Ass essing t he ‘Goodn ess’ of Agi le Methods 44 ## " Accommodat ing Cha nge X Strongly Very we ll Sufficient ly Somewhat Very l ittle Continu al st akehold er commun icati on and colla borati on X Strongly Very we ll Sufficient ly Somewhat Very l ittle Frequent Ref lection and Improvement X Strongly Very we ll Sufficient ly Somewhat Very l ittle Striving f or Customer Satis faction X Strongly Very we ll Sufficient ly Somewhat Very l ittle By def inition, XP value s ‘Si mplicit y’ and ‘Ac comoda ting ch ange’ . From Table 4.8 , we know that XP is maxima lly adequat e with resp ect to the said princip les. The method also values effect ive communicat ion and continuous feedback. Iterat ions are client - driven and teams strive to maximiz e customer satisfaction (Table 4.8). Summary XP bes t refl ects the agi le philosop hy. It is des igned for small teams. From Table s 4.7 and 4. 8 , we kno w that X P su pports its state d obje ctives and p rinciples to a large extent. The method is designed to accommo date change even late in the development lifecycle. Based on our assessment, we conjecture that XP is the most adeq uate with re spect to its to uted objectiv es whe n comp ared to Method A and FDD. 4.3.3 Method A Adequacy Ass essment Organi zation X is a Fortun e 1000 company that provides software development, agile coaching and IT consulting services . The m ethod under consideration in this sub - section, Method A is a proprietary agile metho d of Organiza tion X. As menti oned previou sly, Method A is an instan tiat ion of XP. It embodies almost all of the values and principles stated by XP. H owever, the culture of the organization precludes the ad option of certa in toute d princ iples a nd pra ctices. A form er e mployee of Organiza tion X wh o had Section 4 – Our Propose d Approach to Ass essing t he ‘Goodn ess’ of Agi le Methods 45 #$ " adopted Method A for over two y ears provided us with an overview of the method. Given the OPP Framework as a baseli ne, the following are so me of our observatio ns. Observ atio ns ab out Method A Wit h r esp ect to t he objec ti ves , Me thod A prim ari ly fo cuses on be ing h uman - centric and delivering value to all the stakeholders involved. Method A touts continuous learning and innovation as one of its objectives but, in reality, th e teams d o not supp ort this objec tive. Great emphasis is plac ed on deli veri ng workin g softwa re at reg ular i nterv als, ma intaining a constant development pace, continual stakeholder comm unication and collaboration, and striving to satisfy the customers. S implicity and frequent reflection and learni ng are principles that have the least support. XP mandates that customer s be co - locate d in o rder to ac hieve face - to - face communication and collaboration at any given t ime. How ever, in reality, co - location though preferre d, is not always feasib le. H ence, to o vercome this issue, Meth od A stipulates that custom ers be present during th e first and last days of an iteration. Communicat ion with the customer is achi eved via other means such as video conferencing during other times. Meth od A mandat es the adop tion of Cont inuous Inte gra tio n whic h is “ a software development practice where m emb ers of a team integrate their work frequently, usually each person integrates at least daily - leading to multip le integra tions pe r day” [ 10 ] . Also , “each integ ration is verified by an automa ted bu ild (includ ing test) to dete ct integration error s as quickly as possible. Many team s find that this ap proach leads to significantly redu ced integration problems and allows a te am to develop cohesive software more rapidly.” [ 10 ]. Customer / U ser Acceptance Testin g (CAT/ UAT) is perfo rmed on t he last day of an iteration. The custom ers / users test the sy stem built thus fa r and dete rmine if th e acceptance criteria have been met. Bug fixes and enhancement s to the s ystem are classi fied as ne w tasks and are performed immed iately. Adopti ng Test Dr iven Dev elopme nt (T DD) is option al. H ence, the team m embe rs m ay or may n ot w rite the t ests before wri ting cod e. Section 4 – Our Propose d Approach to Ass essing t he ‘Goodn ess’ of Agi le Methods 46 #% " The teams are not self - organizi ng. The management plays a key r ole in creati ng the teams for various projects. Adequacy Assessment Table 4.9 provides a summary of the linkage coverage and weighted coverage values for the objectives and principles touted by Method A. Table 4. 9 . Linkage coverage fo r Method A Objectives Linkage Coverage Weighted Coverage Human - centric 4/ 4 6/ 6 Value - driven 5/ 6 7/ 8 Mini mal Waste 4/4 5/5 Maxim al Adapt abilit y 4/ 5 5/ 6 Principles Linkage Count Frequent Deli very of Worki ng Software 8/ 8 Technical Excelle nce 15/ 18 Simplicity 5/ 6 Empowering t eams of mot ivated i ndividual s 4/ 5 Co nstant Development Pac e 10/ 10 Accommodat ing Change 11/ 11 Continu al sta keholde r communic ation and collaboration 6/7 Striving for Customer Satisfacti on 4/ 5 We m ap the link age c over ag e values presented in Table 4.9 to the Lik ert scale tha t we have be en using for our assessment purposes (Section 4.1). The resul ting Liker t scale va lues are provid es in Tables 4.10 and 4.11 . Section 4 – Our Propose d Approach to Ass essing t he ‘Goodn ess’ of Agi le Methods 47 #& " Table 4.10 . Adequacy assessm ent of Method A with respect to its touted objectives (using weighted l i nkage count) Human - centric X Strongly Very we ll Sufficient ly Somewhat Very l ittle Value - drive n X Strongly Very we ll Sufficient ly Somewhat Very l ittle Mini mal Waste X Strongly Very we ll Sufficient ly Somewhat Very l ittle Maxi mal Adapt abil ity X Stron gly Very wel l Sufficient ly Somewhat Very l ittle As shown in Tab le 4. 10 , Metho d A, like X P, m axim ally sup ports all of its state d ob jectives. By design, Meth od A does not state ‘Conti nuous Innov ati on and Lear nin g’. As disc usse d previ ously , withi n t he organization, the definition of the adopted method differs from the m ethod in its original form. Hence, Meth od A, which i s an inst ant iat ion of XP, does no t stat e cer tai n objec tiv es and princi ples defi ned in the baseline method. Table 4.11 . Adequacy assessme nt of Method A w ith respect to its state d principles (using linkage count) Frequent de livery o f working s oftware X Strongly Very we ll Sufficient ly Somewhat Very l ittle Technica l Excel lence X Strongly Very we ll Sufficient ly Somewhat Very l ittle Simplicity X Strongly Very we ll Sufficient ly Somewhat Very l ittle Section 4 – Our Propose d Approach to Ass essing t he ‘Goodn ess’ of Agi le Methods 48 #' " Empowering teams of M otivat ed Indiv iduals X Strongly Very we ll Sufficient ly Somewhat Very l ittle Constant development pace X Strongly Very we ll Sufficient ly Somewhat Very l ittle Accommodat ing Cha nge X Strongly Very we ll Sufficient ly Somewhat Very l ittle Continu al st akehold er commun icati on and colla borati on X Strongly Very we ll Sufficient ly Somewhat Very l ittle Striving f or Customer Satis faction X Strongly Very we ll Sufficient ly Somewhat Very l ittle Meth od A does not supp ort ‘Freq uen t Refl ectio n and Lea rni ng’ . The oth er stat ed pri ncipl es are supported maxima lly (Table 4.11 ). Method A foc uses on delivering wo rking software fr equently, maintaining a constant development pace, facilitating continual stakeholder com munication and collaboration, and striving to satisfy the customers as shown in Table 4.11 . Summary Meth od A is a n i nstan ce of XP . Its appli catio n wit hin a n o rga niz at ion precludes the adoptio n and use of certain principles and practices. C omparing the adequacy assessment results of XP (Tables 4.7, 4.8, an d 4.9) and Method A (Tables 4.9, 4.10, and 4.11 ), we c an observe the d ifferences in their stated objectives and principles. M ethod A does not support the objective ‘Continuous Innov ation and Learning’ and its associated principle ‘Frequent Reflection an d Improvement’. The other touted objectives and principles are maximally achi eved. In assessing the adequacy of XP, FDD , an d Metho d A, we recognize that Meth od A with resp ect to its stat ed objec ti ves is (1) le ss ade quate than XP and (2) mor e adeq uat e tha n Section 4 – Our Propose d Approach to Ass essing t he ‘Goodn ess’ of Agi le Methods 49 #( " FDD. That is, it falls in between X P and FDD with respect to supporting its stated objectives and principles. 4.4. Proposed Work Current ly, we have designed the OPP Framework to guide our approach to assessing adequacy, capability, and effectiveness. The tasks identif ied next have to be completed with respect to the assessment approach. 1. Assessing Adequacy: We f oll ow a t op - down approach in traversing the linkages between the objectives, principles, and practices t o assess adequacy. To evaluate the adequacy of any given agil e method, we st ill need t o ! define the final criteria f or mapp ing the linka ge covera ge values to a Li kert scale. ! assign wei ghts to lin kages between p rinci ples and pra ctic es ! revisit the adequa cy asse ssment results after assignin g w eights to the linkage s betw een p rinciples and practices and calculat e the values again. We also pla n to assess the a dequacy of at least two more agile methods. Also, we propose to rank the metho ds based on t heir adequacy . Such a ran king scheme wo uld help us assess the exten t to whi ch ea ch metho d su pport s its stated objecti ves. 2. Assessing Capabil ity and Effecti veness: Assess i ng the capability of an organization to support the implementation of an agile method, and the effectiveness of that m ethod, involve a top - down and bottom - up traversal of the linkages established between the objectives, principles, and practices. Presently, we have a skeletal structure of our approach to asses sing cap ability an d effec tiveness. To de fine an d imp lement our ap proach, the follo wing tasks ha ve to be com pleted. ! We need to identify observable characteristics of the people, process, project and product associated with each practice. The primary difference in the tw o assessment perspectives is the types of in dicators used. We use peop le, pro cess, a nd p roject in dicators to assess capability and process artifacts and project indicators to assess effec tiveness. A s discus sed previo usly, each indicator is a (practice, property) pair. See A ppendix C for a prelim inary list of observable characteristics. Section 4 – Our Propose d Approach to Ass essing t he ‘Goodn ess’ of Agi le Methods 50 $) " ! We nee d to define metrics for the evaluat ion of each indicat or. The valu es prod uced by the metri cs are t hen utilized in computing the extent to which each practice is used by an organization. ! The m etric s may produce values that are subjective , objective, numerical, binary, etc. These values have to be aggregated in order to ascertain the degree of usage of a practice. Hence, we need an approach to map the varyi ng types of va lues produced by the metrics t o a uniform sca le. ! Current ly, we do not possess a definiti ve approach to represen ting the resul ts obtained when assessing the adequacy of an agile m ethod, t he capability of an organization in supporting the implem entation of that metho d, and the method’ s e ffectivene ss. We need to describ e a m ethod that wou ld enable u s to represen t the results ob tained. Though we have outlined the approach to assessing the ca pability and effectiveness, we first need to identify propertie s of the people, proces s, proj ect, and prod uct as sociated with the practices, and the appropriate metrics , in order to effect a bottom - up t raversal of the linkages to assess agile methods from the said per spectives. References [1] J. Highsmith, Agile Soft ware Development Ecosystems : Addison - Wesley, 2002. [2] P. Abrahamsson , et al. , Agile Software Devel opment Methods: Review and Analys is . Finland: VTT P ublications, 2002. [3] R. Jeffr ies. ( 2001). What is Extre me Pro gramming . Available: http://xprogramming.com/xpmag/whatis xp.htm [4] M. Pop pendi eck a nd T. Poppe ndiec k, Lean Software Developmen t: An Agile Too lkit : Addison Wesley, 2003. [5] A. Cockbu rn, Crys tal Clear : A Human - Powered Met hodology for Small Teams : Addison - Wesley Prof essi onal, 2004 . [6] S. R. Palmer a nd J. M. Felsin g, A Practi cal Guide to Feature Drive n Development . Upper Saddle River, Ne w Jersey: Prentice Hall PTR, 2002. [7] O. Balci , e t al. , "Credibility ass essment: a collabo rative evaluation en vironment for c redibility assessm ent of modeling and simulation applications," presented at the Proceedings of the 34th conference on Winter simulation: explori ng new f rontiers, San Diego, California, 20 02. [8] M. L. Talb ert, "A Me thodol ogy for t he measure ment and e valuat ion of c omplex syst em de signs ," Ph .D. D isser tati on, Computer Sci ence, Vir ginia Tec h, Blacksb urg, 1995. [9] A. Qumer and B. He nderson - Sel lers, "An eval ua tion of the degree of agility in six agile m ethods and its app licability for method engineering," Inf. Softw. Techno l., vol. 50, pp. 280 - 295, 2008. [10] M. Fow ler. (200 6). Co ntinuous I ntegration . Available: http://www.martinfowler.com/arti cles/continuousIntegrati on.html Section 4 – Our Propose d Approach to Ass essing t he ‘Goodn ess’ of Agi le Methods 51 $* " Selected Bibliography [1] P. Abrahamsson , et al. , Agile Software Devel opment Methods: Review and Analys is . Finland: VTT P ublications, 2002. [2] J. Highsmit h, Agile Sof tware Development Ecosystems : Addison - Wesley , 200 2. [3] S. R. Palmer a nd J. M. Felsin g, A Practi cal Guide to Feature Drive n Development . Upper Saddle River, Ne w Jersey: Prentice Hall PTR, 2002. [4] A. Qumer and B. He nderson - Sel lers, "An eval uat ion of the degree of a gility in six agile me thods and its applic ability for method engineering," Inf. Softw. Techno l., vol. 50, pp. 280 - 295, 2008. [5] A. Koch, Agile Software Development : Evaluatin g the Methods for Your Organi zation Art ech House Publishe rs , 2004. [6] B. Boehm an d R. Turne r, Balancing Agility a nd Discipline : A Guide f or the Perpl exed : Addison - Wesley, 2004. [7] http://www.featuredr ivendevelopment.com/ [8] R. Jeffr ies. ( 2001). What is E xtreme Programming . Available: http://xprogramming.com/xpmag/whatisxp.htm [9] K. Beck and C. An dres, Extreme Programming Explained: Embra ce Change , 2 ed.: Addison - Wesley Pr ofession al, 2004. Section 5 – Substantiating t he OPP Framework 52 $+ " 5. Substantiating the OPP Fr amework The OPP Framework gui des the ass essment pro cess. Our goa l is to sub stanti ate both t he components of the OPP Framew ork and o ur process fo r assessing a dequacy, c apability an d effectiven ess. The rese arch outlined in this secti on is work in progress. 5.1. Substantiating the components of the OPP Framework The objectiv es, princi ples, practices, and the linkages between them, form the core of the OPP Framework. The Framework also identifi es people, process, project and product ch aracterist ics that are necessary to assess the capability of the organization and the effectiveness of the method under consideration. W e outline the following 2 - step ap proach to substantiating the components of the OPP Framework: Step 1 : Gat heri ng evi d ence from literature and other sources t o ! Substantiate the linkages between objectives , principles and practices We are curr ent ly inv olved in gat her ing evi dence fr om resea rch paper s, exp eri ence repo rts , whit e papers and books to valida te the existence of the proposed linkages. Our effort s in research, learning a nd intera ctions w ith mem bers of the agile comm unity pr ovide us with the necessa ry evidence for the existence of the identified linkages. We have currently substantiated approximately 50% of the ident ified linkag es. ! Substantiate the weights ass igned to the linkages between the objective s, principl es, and practi ces We rea li ze that for each objec tiv e a nd its set of assoc iated pr incip les, some princ iples may sup por t that objectiv e to a greate r extent tha n the others. Hence, we have assigned weights to the linkages between the objectives and principles. We are currently in the process of apportioning weights to the linkage s betwee n the princ iples and p ractices. W e plan to re quest subje ct matter e xperts to evaluate our weights distr ibution scheme and provide feedback. ! Confir m the indic ators We ar e in t he proc ess of id entif ying al l the (p racti ce, pr ope rty ) pa irs tha t fo rm th e in dic ato rs. We then propose to substa ntiate the in dicators by gatherin g e vidence f ro m litera ture an d discu ssions with members of the agi le c ommunit y. Section 5 – Substantiating t he OPP Framework 53 $! " Step 2: Valida te the component s of the OPP Framewor k by obtai ning feedback from agil e practi tioners through p resentation s and using survey instr uments. We propos e to ga the r fe edba ck fr om mem bers of the agile community about t he OPP Framework and its utility . W e ha ve id entified four target organ izations aroun d the Blac ksburg, VA are a w here we intend to present o ur wor k and re quest the practitione rs to com plete a survey th at wou ld provide us with feedback. The survey questions would focus on the utility of the OP P Framework, its components and the assessm ent proc ess. We have spok en to repre senta tives from the tar get org ani zat ions and they have been recep tive to participati ng in our subs tan tiation efforts. 5.2. Substantiating the assessment process We addr ess the asse ssme nt of agil e met hods from thre e pers pecti ves – adequacy, capability and effectiveness. We real ize that in order to effect ively substantiate our assessment approach, the OPP Framework has t o be appli ed within an organi zation. We envisio n a two - step process for substantiating and validating the assess ment approach. Step 1: Using the OPP Framework , we first propose to assess the adequa cy of multi ple agil e metho dolog ies endorsed by the agile community (XP, Scrum, Lean, etc.). Current ly, we have assessed the adequacy of XP, FDD and Method A. Method A is an in stanti ation of XP withi n an organiza tion. Our preli minary assessmen t result s indicat e that XP is more adequate than FDD and Method A wit h respe ct t o its s tated o bjecti ves. Also, FDD i s th e le ast adeq uate wit h re spec t to its touted ob jectives. We p ropos e t o ass ess t he ad equacy of t wo ot her s tanda lon e me thods na mely L ean, and Cr ystal . Step 2 : Secondly, w e intend to a pply the O PP Framework wit hin multi ple organizati ons to as sess 1. the adequacy of its agil e method and 2. the capability of the organization to provide the supporting environment to implement that agile metho d Gather ing f eedbac k f rom pra ctit ioners is an e ffor t to subst ant iate our approach and assessing t he ‘goodness’ of ag ile methods adop ted by organizations is a validation process . In addition to gath ering feedback about our assess ment approac h, we propos e to assess the adeq uacy of agile methods ad opted by o ne or m ore of our target organizations and capability of those organizations to suppor t the imple mentation of the said method s. We inte nd to interv iew proje ct manag ers and re quest Section 5 – Substantiating t he OPP Framework 54 $# " that the y give us a walk through of th eir pro cess. W e also plan to o bserve one o r mo re t eams in action to understand their approach to developing software. This w ould provide us with key insights about the process followed and the supporting environment, and thus facilitat ing the assessment of adequacy of the meth od and ca pability of th e orga nization. After assessing the adequacy of the agile methods and the capa bility of the targ et organ izations, w e prop ose to gather feedback about the as sessmen ts to attest to the validity of our resu lts. Whil e vali dat ing our ap proac h to as sessi ng the ef fecti veness of an agile method is necessary, it requires a longitudinal study t hat falls beyond the scope of our immediate validation goals. Section 6 – Conclusion " 55 $$ " 6. Conclusion Our resear ch is motivate d by the need for a compre hensi ve approach to asses s the ‘goodness ’ of agil e me thods. W e a ssess ‘g oodness ’ of an a gile m ethod based on ( a) its adequac y, (b) the capability of an organization to provide the supporting environment for implementing the method, and (c) the effectiveness o f that method. To guide our assessment, we propose th e OP P Fra mewo rk. The Fram ework identifies obj ectives, princip les, practices an d properties, an d linkages b etween the m to suppo rt the assessment process. Currently, we have ! Desig ned t he OPP Fra mework, ! Identified objectives, princ iples, and practices, an d determined the linkages between them, ! Defin ed t he ap proach to assessin g ade quacy, capabil ity, a nd ef fect ivene ss, ! Outli ned the necessary subs tant iati on effor ts, and ! Assess ed the a dequac y of thr ee a gile metho ds – FDD, XP and Method A. We have dete rmi ned th e linkages between the objectives, principles, and practices based on reasoned conjecture. In order to establish their co mpleteness and validity, we propose to substantiate them by gathering evidence from literature and feedback from agile practitioner s. Current w ork is underway to confirm the defined linkages. To assess cap ability and effectiveness, w e still need to identify p eople, process, proje ct, and product pr operties associ ated with the prac tices. For each (practice, pr operty) pair, or indicator, we propose to define m etrics for assessing the extent to which the said practices are being used. The values obtained will be propagated in a bottom - up fashion to determine the capability of an organization to support the m ethod’s stated objectives, and the e ffectiveness of the m ethod itself. Our proposed subs tantiation approach includes a study of one or more organizations to assess the ‘goodness’ of their agile methods, at least from an adequacy and capability perspective. We recognize that the Framework wo uld e volve based on f uture research considerations. Appe ndix A – Worki ng de finit ions 56 $% " Appendix A Here we provide the worki ng defi niti ons (Table A.1) for the object ives and prin cipl es identif ied as a part of the OPP Framework. These definitions have stemmed from our understanding and in - depth analysis of the agile ph ilosophy a nd metho ds currently used. Table A.1 . Working definitions for the objectives and prin ciples Object ives Human – centric People are more important than proc esses, pra ctices and tools Value - driven Maxim ize value to all the stakeh olders. Value may take the form of in creased revenue, improved custo mer satisfaction, etc. Minim al Wa ste Keep t hings simple. Bu ild o nly wha t is neces sary. Maxim al A dapt abil ity Flexibili ty, ability to accommodate change and having the freedom to choose practices with respect to the p eople, process, project , and product. Continuo us innov ation and lear ning Frequently reflect o n the proce ss and lear n from past mistakes t o innovate and improve. Principl es Frequent deliv ery of working software Delive r workin g soft ware frequ ently; ite ratio n len gth – 2 to 4 weeks. Technical E xcellenc e Excellenc e of the people, proc ess and practic es – selecting the right people, right process and right practices to build working software of value to the customer. Technica l ex cellence is also con veyed through the suppo rting environment. Simplicity Of th e proc ess a nd pro duct – keep the process simple and produce a product cont aining only what i s necessary. Empowering teams of Motiv ated In dividual s Build teams of motivated in dividuals and empower them; the decision - making proces s is taken to the lowest level. Constant devel opment Pac e Build software at a consta nt pac e; the amount of work achiev ed duri ng each iter ation should be cons tant. Appe ndix A – Worki ng de finit ions 57 $& " Accommodati ng Change Accommodat e chang e with minimal impact. Continua l stak eholder communication an d coll aboratio n The stake holders interact and work together at reg ular int ervals to disc uss the system bei ng built . Frequent Reflec tion an d Improve ment Revisit the proces s regul arly; retro specti ve meet ings help in l earning more about t he existing proc ess and support improvem ent. Striving for Customer Sat isfactio n The main o bjective is to satisf y and pr ovide maximum v alue to the cust om er. " Appendix B – Identified linkages between objectives, princ iples, an d practices 58 Appendix B Tables B.1 an d B.2 provide an alterna te repre sentati on of the identified linkages b etween objectives and principles, and principles and practices respectively. An “X” mark or a number within square brackets in a table cell denote the existence of a linkage. Our approach has been to gather evidence fr om literature and other sources to establish the validity of some of the linkages. The numbers represented w ithin square bra cket s denote the reference number of the source from which the evidence can be found. The references can be found at the end of thi s section. Th e cells with the “X” mark r epresent li nkages tha t ha ve no t be en su bstan tiat ed yet. Table B.1 . Identified linkages betw een objectives and principles Object ives/ Principles Freq. Deliv ery Tech. Excelle nce Simplicity Emp. Teams of motiva ted individuals Const. Develo pment Pace Acc. Change Continua l stkhlder comm. and coll. Freq. r efln and learning Striving for cust. satisfaction Human - centric [7][18] [1][7] X [7][18] Value - driven X [11] X X X X Mini mal wast e X X X X Maxim al Adapt a bility X X X X X Continu ous Inno vation and Learning X X Table B.2 . Identified linkages betw een principles an d practices Practice s/Princip les Freq. Deliv ery Tech. Excelle nce Simplicity Emp. Teams of motivated individuals Const. Develo pment Pac e Acc. Cha nge Continua l stkhlder comm. and coll. Freq. r efln and learning Striving for cust. satisfaction Iterative and Incremental De vt. X [8] [5] X Evolutio nary Reqts . X X X [16] [15] X Refacto ring [24] X TDD [28] [28] [15] Aut omated test builds X X Pair Progr amming [10][14] [7][10] [10][14] Appendix B – Identified linkages between objectives, princ iples, an d practices 59 Mini mal BRUF, and BDUF X X JIT X X X Self - Organi zing teams [12][25][27] X Worki ng env iron ment reflective of agile philosophy X X Continu ous Del ivery X X X [27] Constant Velocity X X X Colloca ted cu stomers [6] [25] [25] [25] X Continu ous Feed back [6] [5] [5][12] [3][12] [8] [6] X Prioriti zation X X X Daily Progress Tracking Meetings X X X [22] Agile Documentatio n X X X Agile Project Estimati on X X Frequent fa ce - to - face communication [6] X X X X Retrosp ective meeti ngs X [23] Client - driven iterations X X Smaller and frequent releases [1] [8][12][17] [5][12] Appropr iate compo sition of expertise [19][20] [19][20] Practice s/Princip les Freq. Deliv ery Tech. Excelle nce Simplicity Emp. Teams of motivated individuals Const. Develo pment Pace Acc. Cha nge Continua l stkhlder comm. and coll. Freq. r efln and learning Striving for c ust. satisfaction Appendix B – Identified linkages between objectives, princ iples, an d practices 60 Software Configur ation Manag ement X Code Owner ship (collective and individual) X Iteration Progress tracking an d reporting X X Adheren ce to codi ng standards X Practice s/Princip les Freq. Deliv ery Tech. E xcellence Simplicity Emp. Teams of motivated individuals Const. Develo pment Pace Acc. Cha nge Continua l stkhlder comm. and coll. Freq. r efln and learning Striving for cust. satisfaction Acknowled gement I thank Vinh To for hi s help in substantiating the li nkages presented in this section . Evid ence from literature for th e substantiated lin kages p resented in tables B .1 and B.2 . [1] M., Mirakho rli, A., Rad, F., Aliee et al. "RDP Technique: a Pra ctice to Customize X P". APSO'08 . 2008. [2] J., Ryan, R., Scudiere. "T he Price of Agile is Etern al Vigilance". Agile Co nference. 2008. [3] T., Dingsoyr, G., H anssen. "Extendin g Agile Method s: Postmortem R eviews as Extend ed Feedback". L SO 2002, LN CS 2640, pp.4 - 12, 2003. [4] P., Grisham, D., P erry. "Customer R elationsh ip and Ex treme Prog ramming ". HSSE 2005. [5] Poppendiec k, M. and Popp endieck, T. "Lean S oftware Develo pment: An Ag ile Toolkit." Addison - Wesley , Bo ston, MA, 2003 . [6] K., Petersen, C., W ohlin. "A comp arison of issues and ad vantages in agile and in cremental development between stat e of the art and an industri al case". Journal of System and Software, 2009. [7] M., Levy. "Kn owledge M anagement in Pra ctice: The Case of A gile Software Dev elopment". CH ASE'09. 2009 . Appendix B – Identified linkages between objectives, princ iples, an d practices 61 [8] Karlström, D., Ru neson, P. "Com binin g Agile Methods with St age - Gate Pr oject Manageme nt". Softwa re IEEE , vol ume 22. [9] Abrahams son, P., Koskela, J. "Extre me programm ing: a survey of em pirical data from a co ntrolled case study". ISE SE 2004. [10] Williams, L., K essler, R. et al. "Strengtheni ng the Case for Pair - Progra mming". IEEE Softwar e 2000. [11] Maruping , L., Venkatesh, V., A garwal, R. "A Con trol Theory Perspe ctive on Agile Me thodology Use an d Changing U ser Requir ements". Information Systems Resear ch, Vol 20 Issue 3, 2 009. [12] V idgen , R., Wang , X. "Coev olving Sys tems and th e Organiza tion of Agile Software Developm ent". Infor mation Sy stems Rese arch, Vol 20 Issue 3 , 2009. [13] Conboy, K . "Agility from First Princ iples: Reconstructing th e Concept of Ag ility in Information Syste ms D evelopment". Information Systems Research, Vol 20 Issu e 3, 2009. [14] Dawand e, M., Johar, M., Ku mar, S., Mooke rjee, V. "A Comp arison of Pair Versu s Solo Programm ing Under D ifferent Objective s: An Analytical Approac h". I nformat ion Systems Research, Vol 19 Issue 3, 2008. [15] Ji, Y., Mooke rjee, V., Sethi, S. "Optima l Software Develo pment: A Con trol Theoretic Appro ach". Information S ystems Research, Vol 16 N o 3, 2005. [16] Cao, L., Ram esh, B. "Agile Req uirements Engine ering Practices: An E mp irical Study". I EEE Sof tware 200 8. [17] Hodgetts, P. "R efactoring the Dev elopment Proce ss: Experiences with th e Incremental Ad option of Agile Practice s". Agile Deve lopment Confe rence, 2004. [18] Shore. J and S . Warden, The art of agile devel opment . Beijing ; Seba stopol, CA : O'Reilly M edia, Inc., 2 008, pp 36 1 - 363. [19] A. Cockbu rn, Agile S oftware Dev elopment : Addison - Wesley , 20 02, pp 14 - 15. [20] Boehm. B and R. Turner, Balancing Agil ity and Discipli ne: A Gui de for t he Perple xed : Addiso n - Wesle y, 2004 , p p 15 6 - 160 [21] Weyrauc h, K. "What are w e arguing about? A framework fo r defining Agile in our O rganization.". Procee dings of AGILE 2006 Confere nce (AGIL E' 06). [22] Lindvall, M., B asili, V., Boehm, B. et al. "Em pirical Findings in A gile Methods". X P/Agile Universe 2002, LN CS 2418, pp . 197 - 207. [23] Reifer, D. "Ho w to get the most out o f Extreme Progra mming/Agile M ethods". XP/A gile Universe 200 2, LNCS 241 8, pp.185 - 196, 2002. [24] Tingling, P., Saee d, A. "Extreme Pro gramming in A ction: A Longitud inal Case Study". HCII 2007 , LNCS 4550, pp.242 - 251, 2007. [25] Cockburn , A., Highsmith, J. "Ag ile software develop ment: the people fac tor". Computer , vol. 34, no . 11, pp. 131 - 133 [26] Cockburn , A., Highsmith, J. "Ag ile software develop ment: the business of innovation". C o mputer , vol. 34, no. 9, pp. 1 20 - 122, Sept. 2001 [27] Paulk, M. "A gile Methodolo gies and Process D iscipline". Institute for Softwa re Research, Carne gie Mellon Univ ersity, 200 2. [28] George, B., W illiams, L. "A structured experiment of test - driven developmen t". Inform ation and So ftware Te chnology , 2004. ! Appendix C – Observable properties of the peo ple, process , project, and product 62 !" # Appendix C As mentio ned earl ier, our approac h to assess ing (i) the capabi lity of an organi zati on to suppor t the instantiation of a n agile meth od, and (ii) th e effec tiveness of the m ethod its elf, invo lves the id entification and use of indicators. An indicator is a practice, property pair. Properties are observable characteristics of the p eople, process, project, and product that a re refle ctive of the environ ment of th e org anization. We have identified a partial list of observab le properties for th e practices that are a p art of the OPP framework. These p roperties w ere d efined by answering the set of five questions discusse d prev iously in Section 3.2. Tabl e C.1 bel ow represent s our initia l lis t of obs ervable p rope rties. This list is not complete. Table C.1 . Observable propertie s of the people, pro cess, project, and p roduct Practice s Observa ble p ropert ies o f the people, process and project for assessing capabili ty Observa ble p ropert ies o f the product and the proce ss artifacts created for assessing effecti veness Iterative and Increm ental Devt. 1. Existence 1. Potentiall y shippable product at the end of e ach iteration/ re lease - cycle Evolutio nary Requir ements 1. Existence of JIT gathering/ refinement of deta ils 2. Feature/ story - driven decomposition of user needs/ requirements 3. Existence of a prioriti zation scheme – customers determine the priorities at the featur e - level an d the develop ers at the story and task level 4. Supporting tool s 1. Well defined and prior itized stories and features 2. JIT refine ment of features a nd stories 3. Features, stories and t asks are validated by cont inuous feedback 4. Features, stories and t asks are of the right leve l of decom position Refacto ring 1. Preserve ext ernal behavior 2 . Main tainability TDD 1. Existence 2. Expertise Automat ed te st bu ilds 1. Tool support Pair Progr amming 1. One computer for two programmers 2. Only one person can write code at a time 3. People willi ng to share knowledge 1. Continuous code r eview 2. Knowledge sharing Mini mal BRUF, and BDUF 1. No full - fledged design documentation JIT 1. JIT refine ment of features a nd stories Self - Organi zing teams 1. Management is not involved in team build ing Appendix C – Observable properties of the peo ple, process , project, and product 63 !$ # Worki ng env iron ment reflective of agile philoso phy 1. Open wor k spaces – war room 2. White boards Continu ous Deli very 1. Small relea ses at regular intervals Constant Velocity 1. Use of tools such as burn - up, burn - down and velocity chart s 1. Constant veloc ity over multipl e iterations Colloca ted cu stomer s 1. Customer is a vailable onsite Continu ous Feed back 1. Stakeholders provide feedback regularly Prioriti zation Daily Progress Tr acking Meet ings 1. Existence 1. Meetings do not exceed 10 to 15 minute s Agile Documentatio n 1. Use of index cards to re cord features and stories 2. Use of a sof tware system to manage documentati on 3. Just enough doc umentation Agile Project Es timat ion 1. Customers and t he development team are involved in the estimatio n process Frequent fa ce - to - face communication 1. Stakeholders meet at regular intervals Retrosp ective m eetin gs 1. At the end of each iterat ion, the stakeholders meet to discuss what could be changed to make the process more effect ive Client - driven iterations 1. Existence Smaller and frequent r eleases 1. Existence 2. Length of each iteration 3. Length of each release - cycle Appropr iate composition of expertise 1. 30% Cockburn levels 2 and 3 people Software Confi guration Manag ement 1. Existence of software tools Code Owner ship ( collect ive and ind ividual) Iteration Progress trackin g and reporting 1. Use of burn - up and bur n - down charts Adheren ce to codi ng sta ndards 1. Existence Practice s Observa ble p ropert ies o f the people, process and project for assessing capabili ty Observa ble p ropert ies o f the product and the proces s artifacts created for assessing effecti veness