Creating, Automating, and Assessing Online Homework in Introductory Statistics and Mathematics Classes

Creatin g, Auto mating , and Assessing Online H omew ork in Introductory Statistics and M athematics Class es Karen Santoro 1 and Roger Bilisoly 1 1 Depart ment of Mathema tic al S cienc es, Cent ral Connect icut State Unive rsit y, 1615 Stanley St, New Britain, CT 06050 - 4010 Abstract Altho ugh text book publ ishe rs offer course manag ement systems, they do so to promote brand loyalty, and while an open source tool such as WeBW orK is promising, it requires administrative and IT buy - in . So supported in p a rt by a College Access C hallenge Grant from the Departm ent of Education , we collaborated with othe r instructors to create online homework sets for three classes: Elementary Algebra, Intermediate Algebra, and Statist ics for Behavior al Sciences I. After expe rimentation, some of these q uestion pools are now created by Mathem atica programs that can generate data sets from specified distributi ons, generate random polynomials that factor in a given way, crea te im age files of histograms, s catterplots , and so forth. These programs produce f iles that can be read by the software package , Respondu s, which then uploads the questio ns into Blackbo ard Learn, the course management system used by the Connecticut State Universi ty system. Finally , we summ arize five c lasses worth of student performanc e data along with lessons learn ed while working on this project. Key Words: Statist ics edu cation, Computer - based asses sment, Course management system s , M athem atica 1. Introduction We bec ame i nte re ste d in deve lop ing onl ine homew ork for stud ent s in the intr od uct ory mathe mati cs and statis tic s classe s from differ ent pers pect ives . The first autho r was teaching a nd trackin g studen ts in develo pmen tal mathe matics in o rder to im prove stu dent outcomes, supported in part by a College Access Challenge Grant. This incl udes outre ach to facu lty and studen ts at high sc hools g eograph ically n ear C entral C onnectic ut State Univer sit y (CCSU) , creating sum mer p rogram s suc h as B ridges to help incom ing freshman succeed in our developmental classes, and assessing the progress of the se students from sem ester to semester. Conseq uentl y, she need ed to have online questions that could b e shared with hig h schools, could be used acro ss classes that used differen t textbooks , and could be made accessible to studen ts between semesters or over the summer even if they were not taking a mat h class. Re sour ces that could be us ed in al l these ways preclud es usin g, for example, an online homework platform provided by a textbook publishe r , so cr eating a problem database that was unrestricted by outside interests bec ame im portant. In his introductory statistics classes, the second author bec ame dissatis fied with the traditional model of teachin g from a te xtbook by just lecturing and wanted to incorporate a more data driven approach, such as the one proposed by Cobb (199 1), one that involved collecting data in class to answer questions of interest to students. Eventually he replaced JSM 2014 - Section on Statistical Education 1787 the textbook with PowerP oint slides and activities, but wanted online exercises tha t would giv e feedbac k to studen ts on thei r abili ty to do key task s that are nee ded when one i s a nalyzing a d ata set such as giving a rough estimate of the correlation given a scatterplot. However, b y not using a textbook, a publisher’s homew ork platform was no longer availab le e v en if it were desired, so creating a problem database again w as a solution . Ev entually , we rea lized that w e were both writin g on line problem s, and w e decided to pool our efforts a nd areas of expertis e . Even with two people, the task of creati ng questions by hand was ti me consuming an d daunting. Moreo ver , beca use stud ent s can and do work toget her, it is best to have online homework wher e each student gets m athematically equival ent question that only varies by the values used. Cours e management s ystems vary, but all allow ques tion pools , but this requires tedious cutt ing and pasting if done by hand. Some systems allow question templates a nd can gen erate random values, b ut Blackbo ard Learn, which is used by CCSU, doe s not have thi s fe ature. One soluti on to this proble m is to use an open access onlin e homework bank such as WeBWor K, which is suppor te d by the Math emat ic al Associ at io n of Ameri ca and is described in Gage et al. (2002). This is an appealing option, but one that requires buy - in from the university’s adm inistration and the Inform ation Technology Department, w hich has not happened yet. Anothe r solution to this problem is to automa te th e p rocess, wh ich is th e o ne we pursu ed. This paper outlines how Mathematic a has been used to generate question pools in a format that is easily uploaded to Black board Learn via a program called Respond us. Altho ugh the autho rs ’ experience s are with these specifi c soft ware package s, course manage ment systems gene rall y allow a person to c reat e a text or Word file that can be uploaded, so t he basic i dea is more widel y applicable . 2. Online Homework The effecti veness of o nline hom ework has been often studied and debated. The literature is mixed o n wheth er or not it is m ore eff ective th an the traditional paper homework that is graded by hand, which , fo r examp le, is d iscussed in Burch and Kuo (2010) and Zerr (2007). Howeve r, from dire ct exper ienc e in te aching m athematics and statistics classes, we know that onli ne homework is better than (1) not assigning any or (2) assigning it but not grading it. M oreover, students like getting imm ediate feedback, and w hen given a choice to do online homework or not, t hey often pick the former as will be s hown below. We see onl ine home wor k as a supp le ment ar y too l for asses sme nt of basic ski ll s and computation. In introductory m athematics and statistics courses, some students struggle with basic tasks like solvi ng a lin ear equation o r com puting a z - score. S uch deficiencies are important to discover quickly, which can easily be done with online homew ork because it is easy to create questions that test exactly these types of specific, simple skills. Using multiple choice o r fill - in - the -b lank question s is lim iting in general, bu t th ey are well suited for testing the lower levels of B loom’s taxonomy for the cognitive domain such as the subject knowledge and comprehension levels as described in Bloom et al. (1956) . This allows the tea cher to spend time on asse ssing classro om activitie s, doing group projects, creating deeper exams that require thought and insight, and so forth. JSM 2014 - Section on Statistical Education 1788 These latter tasks require the higher levels of the taxonomy: applicat ion, analysis, evaluation, and s ynthesi s. Altho ugh creatin g our own pool of questi ons is an ongoi ng task that has required a fair amount of work up to this point, we are already seeing m any payoffs. Ou r questions are customized exactly the way we w ant and are independent of any textboo ks. Students can access these questions at no cost and whenever they desire to do. So they can study over school breaks, or be high school studen ts thinking about coming to CCSU , or u se th ese for review for higher - level courses. In pa rticular, this ability to do community outreach and embedded rem ediation would not be po ssible in the traditional publisher’s homework platforms, and as we collect outcome data, we believe that this will help students come to CCSU better prepar ed and get throug h th eir degree pr ograms quicker. Now that our reasons for wanting o nline hom ework have been outlined, the next sectio ns focus on some of the specific choices that have been made in this project. 2.1 Answer Formats Diffe rent course manag ement systems al low diff eren t typ es of questions. However, all systems have at least these two options: multiple c hoice (M C) and fill in the blank (FITB), which are the types we h ave used. Both of these have pluses and minuses, but knowing the potent ial pitf alls and advant ages allows one to use these w ell. 2.1.1 Multiple Choice There is no doubt that MC has disadvant ages. First, creating good distract or answer s requires teaching e xperience and insight into student misunderstandings of the course mater ial . Second , as noted above , MC a ppli es best only to the lower leve ls of Bloom’s taxonom y: subject kno wledge an d co mpreh ension. Third, MC allows guessing. However, there is a f ourth and more tr oubling drawback. As pointed out in Section 1.1 of Sangwin (2013), for some types of problems , checking potential answers is easier th an solvin g the original problem. For example, solving x 2 – 5x + 6 = 0 by either factori ng or the quadratic formula is hard er than substituting x = 2 or x = 3 into th e equa tion. H owever , checking answers is no t alway s easier tha n a dir ect solutio n . Fo r instan ce, solv ing “w hat is the probability that the sum of tw o dice i s 5” requires knowing how to count up the possibilit ies, but if the answers are given in decimal form, that does not give a student an easy short cut. More ove r, MC does have adva nta ges . Fir st , it is easy to ent er suc h ques ti ons into a computer. Second, compared to FITB, there is no ambiguity in the student’s an sw er. Finally, MC is quite good for checking student misunderstandings, especial ly “ buggy rules, ” a term used in Section 6.10 of Sangw in (2013). F or example, some students do think that (a + b) 2 = a 2 + b 2 , which m akes the righ t hand sid e a good distractor. Given the abo ve pro s and cons, MC i s us eful for detect ing c ommon mis take s tha t students mak e even though it can be limiting . Fortun ately, some of its shortco mings ca n be bypassed by us ing FITB, as di scussed below. 2.1.2 Fill in the Blank FITB is not p anacea because it has on e big disadvantage: there are m ultiple ways to write a correct solution. For example, w hat is ½ - ¼ - ¼ ? Of cours e it is 0, but this could be writ te n 0/4, - 0/4, 0/2, 0, 0., 0.0, etc. Including instructions on what is a permissible JSM 2014 - Section on Statistical Education 1789 a nswer such as “write your solution to two decimal places” reduces the possibilitie s, but 0.00 and .00 both satisfy these instructions, and at some point, long detailed instructions get cumbersome. Mor eover, simple m ath problems can have solutions in m ultip le forms, each of which is defensible. For instance, “ expand ( b + a) 2 ” could be answ ered in the following ways: a 2 + 2ab + b 2 , b 2 + 2ab + a 2 , b 2 + 2ba + a 2 , b 2 + a 2 + 2ba, a 2 + b 2 + 2ab , etc . O ne could try to expla in to in troducto ry - level stude nts h ow to define lexicograp hic order o n mono mials to determin e which one of thes e possibilities is “correct,” but that would be confusing, not helpful. In practice, b oth specifyin g the form of the answer and having the online system give full credit to severa l po ssi ble forms can resolve this problem . An examp le of this is discussed below. Of cours e, FITB do es have ad vanta ges, which are easy to st ate . Firs t, unl ike MC, guessing by st udents is no longer effective. Second, students are more likely to realize when they need help. For our purposes of probing our students’ basic subject knowledge and comprehension, both MC and FITB are valuable, and we have not tried using other forms of questions supported b y Blackb oard Lea rn. Moreo ver, by automating the writing of homew o rk questions, we have been a ble to bu ild up larg e question pools relatively quickly, the details of which are discussed in the next se ction. 3. Automating Question Creation Maki ng ques tio ns by hand is tedio us and the chan ce of making at least one error is high when one makes a questio n bank . However, with the ava ilab ili ty of comput er algebr a systems, it is no t hard to write a program that both generates a question along with it s answer. M oreover, for a M C question, distractors can be made using bu ggy rules, and for FITB, alternat e valid forms of the correct answer can be generated. This project uses Math emat ic a t o cr ea te HTML ve rs ion s of the qu estions, answe rs, and if needed, graph ics. Micr oso ft Word acc ept s HTML as inpu t, whi ch then can be save d a s a Word document. This, in turn, is read into the package, Respondus , (see https://www.respondus.com/ ), which can upload quest ions to a variet y of course managemen t systems, includi ng Blackbo ard Learn. For ease of programming and keeping track of which ty pes of questions have been created, each question template corresponds to exactly one Math emat ic a function. 3.1 MC Example: Estimating the Correlation from a Scatterplot Here is an example where MC is the best option. Given a scatte rplo t with, say, a hu ndred points, no one can make an e stimate accurate to two decimal places, and even one decimal place is challenging. But recognizing the sign of the correlation (positive for upward tr ending val ues, negative for downward trending) and being able to distinguish between r = 0.8 vs. r = 0.2 is useful for the students to l earn. To start the process, t he M athematica program generates the absolute value of a correlation from a Uniform[0.5, 0.8] distr ibution, and its sign is determined by a coin flip. Next, a sa mpl e size is pick ed from a Unifor m[50, 200] dist rib utio n, then a biva ria te normal random sample is generated w ith this correlation and sample size. The actual correlation is computed, which is the correct answ er. Th e distractors are the negative o f the cor rect corre lation, and three additiona l values close to 1, 0, and - 1, respecti vely. JSM 2014 - Section on Statistical Education 1790 One nice feature is the abilit y to create a graphic, save this as an image file, then create HTML that inserts this image into the questio n. Speci fica lly , the scatter plot is saved as a PNG file by the Export[] fu nction to a filename g enerated by th e program such as “correlation0029 .png,” w here the nu mber is ne eded becau se a pool of qu estions is bein g created. Th e HTM L code for inc luding this im age is , which is quite short . Code Sa mple 1 has t he specifi c code for these two actions , where data = {x, y} is the sim ulated bivariate normal values. Figure 1 has the HT ML code generated, and Figure 2 shows how this looks in Blackboar d Learn, which is very simila r to how it appears in Micr oso ft Wor d. out = ListPlot[data, AxesLabel - > {"Variable 1", "Variable 2"}, PlotRange - > {{0, Max[x, y]}, {0, Max[x, y]}}, ImageSize - > 500]; filename = "correlation" <> ToString[questionNumber] <> ".png"; Export[filename, out]; ... WriteString[strexam, " \ n filename <> " \ "> \ n

"]; Code Sample 1: Mathe mati ca code that (1) crea te s a scatt erp lo t and expor ts it as a PNG image file , and (2) prin ts a tag to in clude this im age in th e HTM L file it is w riting. Figure 1: HTML code created by the Mathemat ica program. This can be direct ly opened by Microsoft Word, the results o f which is shown in Figur e 2. JSM 2014 - Section on Statistical Education 1791 Figure 2: A MC que stio n read y to be uploa ded by Re spondu s int o Blac kboar d Learn . The ast eris k in front of answer c indica tes that this is the correc t choice . 3.2 MC Example: Area of a Trapezoid Pro blems involving geometry obviously benefit from a graphic. This example, computing the area of a trapezo id given a p lot of its sides and vertices, is one of a pai r w here the other is computing the perimeter. The image was made by combining two plots: one of the point g rid and th e other of the blue lin es of the trapezoid itself. The MC possibilit ies for both types of question give the correct area and perime ter alo ng w ith a n inco rrect a rea and perimeter. In addition, this example has answ ers that include a square root sign, which requi res special format tin g t o l ook like what students are used to. However, Math emat ic a has a f unc ti on call ed TraditionalForm[] , w hich produces typeset mathe mati cs as an image, which can be used as MC optio ns. Figure 3 shows an example of the HTML form of s uch a question, and Figure 4 shows how this appears. One might argue that t his type of question should be put into a FITB f orma t. How ever, there is no need to d ecide between form ats becaus e both could b e used. For ex ample, a homework set could have the MC version first then the FITB. Although there are sophisticated adaptive systems like ALEKS that use artificial intelligence to determine optimal question sequencing, w ith thought a person with teaching experi ence can make an ordering that probes t he strengths and weaknesses of a student, too. JSM 2014 - Section on Statistical Education 1792 Figure 3: HTML code for the questi on display ed in Figure 4. Note that the four multi ple choice answers are all i mages created by Mathematica’s TraditionalForm[] . Figure 4 : A MC qu esti on a skin g for th e ar ea o f the trapezoid. Choice c is the perimeter, b is a per imeter distractor , and d is an area distractor. JSM 2014 - Section on Statistical Education 1793 3.3 FITB Example: Solving Linear Equations Solving a linear equation is a key skill in remedial mathematics and introduct ory statistics. For example, z - scores an d regression line s are both linear. For students whose math skil ls are poor, having a sequence o f questi ons is valua ble. For e xampl e, one can start a homework set with a linear equation with integer coe fficients and an integer answer. Next could be an equ ation with integer coeffi cients with a rati onal answer, and eventually a student needs to be comfortable w ith decim al coefficients, which will generally have a decimal ans wer. For example, a regressi on li ne for real data generally would have deci mal coef fici ents . Figure 5 g ives an exa mple of a linear equ ation w ith b oth inte ger co efficients and an swer. This can be obtaine d by (1 ) picking the solution first, then (2) solving for one of the constant coefficients. For example, suppose x = 5 is the solution, then solvin g the equation below for c after substituting this value for x gives the desired type of problem. Here c = 8, which i s an i nteger as promised. 3x – 2 = x + c Altho ugh these thre e problems were force d to give an intege r an swer, the student is not told this, so he or she may not en ter the answer as an integ er without a dec imal point. Blackbo ard Learn does not h ave an opt ion to c ompare student input with a regular expression, but it is not hard to list a few alternative answe rs that a student might enter. For example, question 2 in Figure 5 gives full credit for - 4, - 4., - 4.0, and - 4.00. It is true that - 4.00000 is also correct, but in the rare case a student en tered su ch a va lue, he or she is likely to ask why cr edit was n ot given and would star t to use less trai ling zeros. Figure 5 : Three linea r equat ions with bot h int eger c oeffic ients and an intege r sol ution. JSM 2014 - Section on Statistical Education 1794 In developmenta l mathematics, working with rational numbers is important, so having students solve linear equa tions w ith ra tional c oefficien ts and requiring the answe r to be a rational nu mber is usefu l. Here getting rid o f all the de nominators by multiplying throug h by an integer is key. Th e easiest w ay to d o this wo uld be to use the lea st comm on multi ple of all t he denominators, but any com mon m ultiple would suffice. Examples of this type of problem are given in F igure 6. For problem 3, not that th e least commo n multi ple of 6 and 4 is 12, but a stude nt might also use 6*4 = 24 to rati onal ize the fractional coeffici ents, hence there are two answers that are accepted: 9 /14 an d 18/28. Finally, Figure 7 shows ho w the fi rst qu estion in Figur e 6 appear s in Bl ackboard Lea rn. Figure 6: Three linear equation s with rational coefficients that require a rational answer from the student. Figure 7 : How Ques tio n 1 in F igur e 6 appea rs i n Bl ackbo ard Learn . The above examples give an idea of th e types of questions we have created so far. But unless students put effort into doing the online hom ework, it will not have much impact. Up to thi s p oint we have five cl asses wor th o f da ta, whic h we descr ibe in the next sec tio n. JSM 2014 - Section on Statistical Education 1795 4. Student Responses and Assessment In the two sections of Interm ediate Algebra shown in Figure 8 , th ere is a po sitive correlation between the online hom ework grades for the semester and the final course grade. Because the former is used to compute the latter, an upper trend is not surprising, but there is clearly m uch variability about this regression line. However, there are no points b elow the red line, which m eans that no students had a high homew ork score and a low final g rade, wh ich is enco uraging . Figure 8 : This scatterpl ot compares the course grad e to the o nline homework grade for two sec tions of Intermediate Algebra . The d ata show s an upward trend, and note that there are no students b elow the red line. In Introductory Stati stics, the course grade was the avera ge of the best four of the following six items: exam s 1, 2, 3, a nd 4, the final, and all the online ho mew ork combined converted to a percentage. Consequently, a st udent co uld skip the hom ework completely, but Figure 9 shows that on ly two students actually did this, and only two more stopped doing the homework early in the semester . Moreove r, these four data values are in fluencing the regression line such tha t its slope is lower than it w ould be witho ut t hese . Redoing th e regr ession without th ese fo ur stud ents, R 2 is 0.62 92, so the upward trend is quite st rong. JSM 2014 - Section on Statistical Education 1796 Figure 9 : This is t he same type of plot as Figure 8 fo r the th ree sectio ns of Intro ductory Statist ics. The four values on left side look influential on the regress ion and represent students who stop doing online homew ork early in the semester . We are st ill coll ec ti ng and plan to tr ack stude nt s as they progr e ss through CC SU, but ou r initial f indings are encouraging : th e h omew ork is positiv ely correlated with the co urse grade, and in Introductory Stat istics w here students are allowed to ski p the homework, they overw helmin gly choo se not to do so . C reating en ough problems for an entire semester is a deman ding task, a nd we hope one day to join our efforts with one of the open source question pools such as W eBWorK. However, given our current freedom from being lock ed into one pu blisher and the ability to pro vide unf ettered access to both potential and current st udents, we are happy with our initial decision to create our own online homework. Acknowledgements Several peopl e have helped and encouraged us in our efforts. Most of all, we thank Davi d Oy anadel and Jennifer Nicoletti from Th e Instructional Design and Technology Resou rce Center (IDTRC) alon g wit h the ir s taff . JSM 2014 - Section on Statistical Education 1797 References Bloom, B. S., Englehart , M. D., Furst, E. J., Hill, H. W., and Kruthwohl, D. R. (1956). A Taxonomy of Educational Objectives: Handbook I Cogni tive Domain , McKay , New York. Burch, K. and Kuo, Y. (2010). “Traditi onal vs. Online Homework in College Algebra,” Mathe mati cs & C omput er Educ ati on , 44(1), 53 - 63. Cobb, G. W. (1991) . “Teachi ng Stati stics : More Data, Less Lecturing .” Amstat N ews , December 199 1, 1 - 4. Gage, M., Pizer, A., and Roth, V. (2002). “WeBWorK: Generat ing, Deliver ing, and Checkin g Math Homework via the Inter net, ” ICTM2 Inte rnational Congress for Teaching of Mathematic s at t he Undergradu ate Level , Herson issos, Cr ete, Gre e ce. Sangwin, C. ( 2013). Computer Aided Assessment of Mathemat ics , Oxford Univer sity Press, Oxford, U.K. Zerr, R. (2007). “A Quantitati ve and Q ualita tive Analysis of the Effectiv eness of Online Homework in First - Semester Calculus,” Journal of Computers in M athematics & Science Teaching , 26(1), 55 - 73. JSM 2014 - Section on Statistical Education 1798