Math Achievements and WeBWorK

Now that WeBWorK version 2.5.1.3 is making its way onto more and more servers, I (being Geoff Goehle from Western Carolina University) wanted to write a bit about a new feature available to professors that want to add a bit of fun to the homework process.  WeBWorK instructors now have the ability to create and award "Math Achievements" and "Math Levels" to students for solving homework problems and for practicing good WeBWorK behavior.  In a nutshell, students can earn achievements by meeting preset goals.  For example, they might earn an achievement for solving 3 homework problems in a row without any incorrect submissions, or for solving a problem after taking an 8 hour break.  Earning achievements and solving problems earns students points and after a student gets enough points they will be given a new "Math Level".  After the break I will talk a bit more about what the achievements are and go into detail about how to enable them, modify them, and even create your own achievements.


As I said, achievements are earned by meeting certain goals.  For example, in the following table I have a partial list of achievements available to students.

As you can see there are a variety of achievements, from the relatively mundane like "Perfect 10" to the more exotic like "Night Owl".  When students earn one of these achievements they are given a notification like the one shown below.

Earning achievements and solving problems in WeBWorK gives the students points.  After students earn enough points they are rewarded with a new level, like the one shown above.


 Students can see what achievements are available and check out their progress to the next level by visiting the Achievements link in the Main Menu. (When achievements are enabled this link will be placed under the Grades link.)  A partial screen shot of the Achievements page is shown below.

One of the additional features of this page is that students can click on the button Enable Facebook Integration to integrate the Math Achievements with Facebook.  What this does is give students the opportunity to post a message to their timeline whenever they earn a new achievement or level.   Update:  Facebook integration is still possible, however, because of the way Facebook works its much more complicated to set up. In general these systems are intended to add an element of play and positive reinforcement to WeBWorK. They are actually an example of a larger trend known as gamification.  I won't talk much about the theory behind gamification or the philosophy of this particular implementation here.  Viewers who are interested in learning more about these things should consider reading my paper Gamification and Web Based Homework which is published in the journal PRIMUS.   This paper can be found here.  Rather, in the rest of this post we will take a deeper look at how instructors using WeBWorK (version 2.5.1.3 or higher) can enable, modify and create achievements. 

First, to enable the achievement system visit the Course Configuration link at the bottom of the Main Menu.  In the General category there should be two options Enable Course Achievements and Achievement Points Per Problem.  (If these options are not visible then it is likely that you do not have a new enough version of WeBWorK.)  By changing the value of Enable Course Achievements to True you will turn on the achievement system.  The Achievement Points Per Problem option controls how many achievement points are rewarded for every problem students completely solve.  (This is in addition to points earned through achievements.)

Note: By default the Achievement Points Per problem is set to 5.  This value is such that if the default level thresholds are used in a course with approximately 200-230 problems per semester then most students will get to level 9 or 10 by the end of the course.  If you use fewer or more than 200 problems in your courses then you should adjust the Achievement Points Per Problem value in ratio to 200. If you add in additional custom achievements, or change the number of points awarded for achievements then this also affects how quickly students will earn levels.

Once the achievements have been enabled students can see the available achievements, their earned achievements, their current level, their progress to the next level, and enable facebook integration by following the Achievements link in the main menu.  For instructors, managing achievements is done by following the Achievement Editor link under Instructor Tools.  (You may have to reload to see these links.)  By default there will be no achievements listed in the Achievement Editor.  Depending on your set up you may be able to import a  default set of achievements by clicking Import achievements from... and selecting Default_achievements.axp.  (You can also assign the achievements to your students at this time.  Achievements will not be visible to students until they are assigned.)  If you do not have Default_achievements.axp listed in the Import drop down menu then you should download the tar.gz files here and here. The first archive should be uploaded and unpacked using the File Manager into the templates directory. (This is the default directory when you enter File Manager.)  The second should be uploaded and unpacked into the html directory. (This is can be found by clicking on the ^ button to go up a directory from templates.)  Once you have done this you should have the ability to import the default achievements as described above.


After you have imported these achievements and assigned them to your students, either during import or by using the appropriate Achievement Editor option, you should be good to go.  With one exception these achievements are intended to run independently of the content of the course.

The one exception to this are the "Challenge" achievements.  These achievements are awarded for completing challenge problems.  If you don't want to have challenge problems then delete or disable these 11 achievements.  If you do want to have challenge problems then create a set called Challenge with 10 problems and the challenge achievements will work automatically. 

If you ever want to take a closer look at which achievements are being earned, or if you are using achievement levels to award extra credit, then you can actually save the student achievement data to a csv file.  This can be done using the scoring option on the Achievement Editor page.  Once you score achievements the results are saved to a file in the scoring folder, which can be accessed from the File Manager after using the ^ button to go up a directory from templates.  Two other things you can do on the Achievement Editor page are to import and export achievements.  Exporting achievements saves all of the achievement metadata to a file in the achievements folder.  To move achievements to a different course you should then create an archive from the achievements folder and extract it into the templates folder of the other class.  The achievement metadata can then be imported as above.  (Note: you may also need to copy over the achievement folder in the html directory to bring the achievement icons along as well.)


Those of you feeling a bit more adventurous might want to start editing the existing achievements, or even creating your own achievements.  In many ways editing achievement data is similar to editing homework data.  If you click the links in the Edit Users column then you will be presented with a list of the students, as well as their individual achievement data.  By checking the appropriate boxes you can change which achievements are assigned to which students and even manually award and revoke achievements from individual users.  (Note: This will appropriately update their achievement score.)   While most of this is self explanatory, you may find that the "Counter" column is largely empty.  This is used in achievements which have a numerical counter associated to them.  For example there is an achievement for finishing 100 homework problems.  For this achievement the Counter column would contain a number indicating how many problems each student has solved.  This value can also be edited and is used to create progress bars which students can see on the Achievement page.


As for editing the global achievement data, this can be done by clicking on the pencil next to the achievement name or by using the "Edit" option in the Achievement Editor page.  You will end up on a page with a table like the one shown below.

This table has a lot going on.  The left most column contains the current icon of each achievement.  The file name for the icon is contained in the third text bar in the right most column.  These icon files are all located in the "html/achievements" directory.  If you put in the wrong file name then you will end up with broken image links.  If you don't put in any filename into the Icon File entry then you will get a default icon.  Next the Edit link in the right most column brings you to the achievement "evaluator" which I will talk about later.  In the second column you have the Achievement ID, which you cannot edit and is not visible to students.  We also have the Achievement Name, which is the visible title attached to the achievement.  The achievement category is in the third text box in the second column and has a special purpose.

Achievements are always sorted (and evaluated) alphabetically, first by category and then by Achievement ID.  This is why the Achievement ID's and categories (usually) start with numbers.  There is also a subtle visual grouping of achievements into categories on the student Achievement page.  Since there are lots of achievements its useful to be able to sort them by "type".  There are two special categories that deserve to be singled out.  Achievements in the "secret" category will not be visible to the students until after they are earned and are for fun/surprise achievements.  There is also a "level" category.  These are the achievements which are used to determine/control a students Math Level.  If you do don't want to use levels then delete or disable these achievements before assigning any data.  If you want to change the thresholds for levels, you can do that by editing the achievement evaluator for the level achievements. 

Moving on, the third column has controls for if the achievement is enabled or not, how many achievement points the achievement is worth, and, if the achievement is the type which uses a counter, the "goal" for that counter.  For example, in the "Perfect 10" achievement for solving 10 problems, the counter field is 10.  In the last column we have the field which contains a description of the conditions necessary to earn the achievement, the name of the achievement evaluator file, and the name of the icon file for the achievement.  By editing this data you can change a lot about how your achievements look and feel, and even modify the activation conditions for counting type achievements. However, in order to really alter how achievements behave, or to make your own, you will need to edit the evaluator files.


Achievements are implemented into webwork as follows.  Every time a student submits a problem WeBWorK goes through the list of assigned achievements for that student.  For each achievement it loads up a perl script, called the achievement evaluator, and runs that script (in a Safe container).  If the script returns 1 then the achievement is awarded and if it returns 0 then the achievement is not awarded.  This means that all of the heavy lifting for assigning achievements is done by these perl script achievement evaluators.  The evaluators are assigned to a particular achievement as described above and the evaluator files are all stored in the achievements directory in the course templates folder.  At heart an achievement evaluator is nothing more than a perl script that returns a true or false value.  For example, here is the evaluator script for an achievement which is awarded when a student finishes a homework set (100% on every problem) between midnight and 2am.

my @timeData = localtime(time);

#test to see if it is between midnight and 2am
#return 0 if it isn't
if ($timeData[2] > 1) {
     return 0;
}

#if it is check to see if we have finished the set
#return 0 if we haven't
foreach my $problemRecord (@setProblems) {
     if ($problemRecord->status != 1) {
          return 0;
     }
}
    
#if we got this far then the student should be awarded the
#achievement
return 1;


As you can see, the script is fairly straightforward.  The only mysterious bit is how it is checking to see if the students have completed their homework.  When these scripts are run there are a variety of environment variables available.  Some of these variables can be modified so that changes to the variables will persist between calls to the evaluator, and some cannot be modified (or rather they can be modified but changes will not be saved anywhere).  I have listed most of the available environment variables below.  Something to note is that many of the variables (like $problem and $set) have the same structure as in the main WeBWorK code.  

• $problem : the problem data (changes to this variable will not be saved!)
  This variable contains the problem data.  It is a hash pointer with the following values (not all values shown)
• $problem->status : the score of the current problem
• $problem->problem_id : the id of the current problem
• $problem->set_id : the id of the set containing the problem
• $problem->num_correct : the number of correct attempts
• $problem->num_incorrect : the number of incorrect attempts
• $problem->last_answer : the last answer submitted (stored in a funky format)
• $problem->max_attempts : the maximum number of allowed attempts
• $set : the set data (changes to this variable will not be saved!)
  This variable contains the set data.  It is a hash pointer with the following values. (not all values shown)
• $set->open_date : when the set was open
• $set->due_date : when the set is due
• @setProblems : the problem data for all the problems from this set. (changes to this variable will not be saved!)
  This is an array of problem hashes.  Each element of the array has the same structure as the $problem variable above
• $counter : the user specific counter associated to this achievement (changes to this variable will be saved!)
  If this achievement has a counter associated to it (i.e. solve 20 problems) then this is where you store the students counter for this achievement.  This variable will initially start as the empty string.
• $maxCounter : the goal for the $counter variable for this achievement (changes to this variable will not be saved!)
  If this achievement has a counter associated to it then this variable contains the goal for the counter.  Your achievement should return 1 when $counter >= $maxCounter.  These two variables are used to show a progress bar for the achievement. 
• $localData : this is a hash which stores data for this user and achievement  (changes to this variable will be saved!)
  This hash will persist from evaluation to evaluation.  You can store whatever you like in here and it can be accessed next time this evaluator is run.  Two things to keep in mind. The data in this hash will not be accessible by other achievements.  If you plan to store something in this hash you have to write code to initialize the data. 
• $globalData : this is a hash which stores data for all achievements  (changes to this variable will be saved!)
  This hash will persist from evaluation to evaluation and, like $localData, you can store whatever you like in here.  This data will be accessible from every achievement and is unique to the user.  Like $localData, you need to initialize any variable you plan on using in this hash.  There are two variables stored in this hash that are maintained by the system. 
• $globalData->{completeSets} : This is the number of sets which the student has earned 100% on.
• $globalData->{completeProblems} : This is the number of problems which the student has earned 100% on.
• Warning: The achievements are always evaluated in the order they are listed on the achievement editor page.  To make matters more complicated, achievements which have already been earned are not evaluated at all.  The up-shot of this is that when modifying variables in $globalData you need to either write your code so it doesnt matter which order the evaluators are run in, or you need to pay very close attention to which evaluators are run and when. 

There are a lot of variables available for use in the evaluator.  What is more, it is relatively easy to change the main WeBWorK code to make more variables available to the evaluators.  The only real limitation is that the variable has to be something already lying around inside WeBWorK somewhere.  Two special variables that deserve to be highlighted are $localData and $globalData.  These variables are special because they persist across calls to the evaluator and are generic hashs so that anything can be stored inside.  For example, what follows is the code to an evaluator which checks to see if the student has solved a problem after taking an 8 hour break.

# Initialize the lastattempttime variable.  
# Note: $localData always starts out empty so often
# you will have to include initialization code.
$localData->{lastattempttime} = time() 
      unless $localData->{lastattempttime};

#Check to see if problem was already previously finished, or if 
#they started a different problem or if the time interval isn't 
#long enough.  If so, reset the lastattempttime, lastset and
#lastproblem variables and return false. Otherwise return true
if ($problem->num_correct > 1 ||
$problem->set_id ne $localData->{lastset} || 
$problem->problem_id ne $localData->{lastproblem} ||
abs($localData->{lastattempttime} - time()) < 28800 ){
        #Note: We don't have to initialize the lastset or  
        # lastproblem hash element because ne will return 
        # true if they are undefined.
        $localData->{lastattempttime} = time();
        $localData->{lastset} = $problem->set_id;
        $localData->{lastproblem} = $problem->problem_id;
return 0;
} else {
return 1;
}

Again the code for the evaluator is fairly short.  In fact most evaluators are even simpler than this one.  A common type of achievement that has not been included in the default achievement package, even though the evaluator files and icon files are both present in their respective achievement folders, are those achievements which reward students for solving specific problems, or a certain number of a certain collection of problems.  For example, the following code awards an achievement for solving a particularly hard optimization problem. 

#List the problem/s which will count toward this achievement 

#as a nested hash, with the set name and problem number as keys

my %validproblems = (

     'SetNameHere' => {

            '1' => 1,

    '3' => 1, }, 

);

#Check and see if this problem was solved for the first time

#and is in the above list of valid problems.  If so return 1.

if ($validproblems{$problem->set_id} &&

    $validproblems{$problem->set_id}{$problem->problem_id} &&

    $problem->status == 1 &&

    $problem->num_correct == 1) {

        return 1; 

}

return 0;   

While the code is simple enough, it will depend on the structure of an individual course.  The instructor needs to go to the evaluator and manually enter which sets and which problems count for this achievement.  This type of evaluator can also be modified, for example, to give achievements for solving 20 derivative problems.  For instructors interested in customizing achievements, these are a good place to start.  They are easy to create and will integrate the achievement system with the content of your course, which will in turn increase its effectiveness. 

In general the open ended nature of the achievement evaluators means that the types of achievements that can be created are really only limited by the instructors imagination and their tolerance for perl code.  Once achievements have been implemented they can be exported and imported using the Achievement Editor menu.  It is the author's hope that there might one day be, for example, school specific achievements which are passed around from class to class.  If you have any questions about the achievement system feel free to email me at goehle@gmail.com.  Good luck with your Math Achievements, I hope the current setup treats you well, and to all you students out there, Happy Cheevo Hunting!