Just-In-Time: Adaptive Homework

Another project that I have been working on for the past couple months is a brand new type of assignment which will eventually join traditional Homework Sets and Gateway Quizzes in WeBWorK.  This new set type is called Just-In-Time and is a kind of adaptive homework; different students will experience the set differently depending on their performance.  However, the usual idea behind adaptive homework is that the homework system will analyze a student's performance and generate or select problems based off of how well they are doing.  Just-In-Time sets work a little differently. For Just-In-Time sets all students will have the same basic set of problems, but if they have trouble they will be given additional problems which will provide "just in time" review of the important concepts.  Below I'll present an example of what a Just-In-Time set might look like and will also talk about how these sets can be built in WeBWorK.  Note:  This feature is currently experimental.  Get in touch with me via email if you are interested in trying it out. 

Lets suppose we are a student starting a Just-In-Time type homework set that is intended to bring us up to speed on some common integration techniques.  When a student starts the set they will see something like this.

One of the options for Just-In-Time sets is to restrict problem progression so that students are forced to do their problems in the order presented.  Here Problem 1 is a u-substitution problem for which the student has 4 attempts.  If they are able to solve the problem within the given number of attempts then Problem 2 will open up and they will move on.  If they do not get the right answer after 4 attempts they will need to do some review. 

In this case the student has not successfully solved the problem.  They are then notified about the fact that this problem has open sub-problems and that they may be able to salvage their grade by getting Problem 1.4 correct.  Of course, they will have work to do before they get to 1.4!  In particular if the student hits "Next" or goes to the main problem set page they will see that there are now four new problems, 1.1 through 1.4, which they did not previously have access to.  These problems will provide review and practice, and then allow the student to retry a problem similar to the original.

This review is not automatically generated or anything like that, it must be created by the course instructor.  However it can be done using the standard PG techniques already in place.  For example Problem 1.1 is a video review of u-stubstitution delivered via YouTube.

Notice we ask a simple question at the end to make sure the student watched the video since we are preventing them from proceeding until they get this problem right.  Once they watch the video, Problem 1.2 presents them with an easy substitution problem, which has unlimited attempts.  For this problem the ShowMeAnother feature has been enabled.  So if a student gets the problem wrong often enough they will have the option to see a different version of the same problem, but with the solution available for study.

The next problem, number 1.3, presents a practice trig u-sub problem for the student to work on.  While a student does have unlimited attempts on this problem, after they get it wrong 3 times they will be given access to 1.3.1.  Problem 1.3.1 has some basic trig derivatives and is further review for those who need it.  However, even if they solve 1.3.1 and practice their trig, the student will still need to go back and do Problem 1.3.  If they are able to solve 1.3 without any trouble then Problem 1.3.1 will never appear.

After finally solving 1.3 the student will then get 4 attempts to solve Problem 1.4 which is similar to, but not the same as, Problem 1.  If they get Problem 1.4 correct then their grade for Problem 1 will be recorded as 100% in the gradebook.  If they get it wrong, and if this feature is enabled, then the instructor will be sent an email letting them know that this student was unable to successfully complete Problem 1, even after going through the review sequence.

Whew!  And this is all for Problem 1.  Problems 2 through 4 would contain similar review structures for other integration techniques which would be unlocked only when students are unable to finish the problem, or after a certain number of attempts.  Their final grade for the assignment will be their adjusted status on Problems 1 through 4.  That is, for each problem their score will be the larger of their score for that problem and their average score from the child problems which count for that problem.  The set score is created from that.

Where do all of these problems come from, you might wonder.  As you might guess, setting up a Just-In-Time set is significantly more work than setting up a regular homework set.  Even an assignment with only 5 top level problems might be expected to have 20 or more child problems to serve as review.  What's more these problems have to work together well and many, like the video tutorial, are unlikely to be found in the OPL.  That being said, problems in Just-In-Time sets are written using PG, just like any problem, so anyone familiar with (or interested in) problem authoring can jump right in.   In general adding problems to Just-In-Time sets is done the same as it is for regular or gateway sets. You add problems using the library browser, or the problem editors, or the "add blank problems" field as you normally would. New problems are always added to the end of the set as a new top level problem, so you will most likely have to reorder them after you add them.  Furthermore importing, exporting and assigning homework sets can all be done using the Homework Sets Editor 2 page, just like any other assignment.

One new tool which has been created for Just-In-Time sets is the Problem Set Detail 2 page.  This can be found by visiting a homework set and then clicking the link on the left hand navigation bar that looks like --Set Name.  You should end up at a page which looks like this.

(Notice the link on the left.)  This portion of the page should look pretty familiar. You set parameters here as you normally would.  Just like any set remember that you have to Save your changes for them to take affect.  The main differences are how you order problems. The problem section of the page looks like this.

Notice the nesting in the problem list. This, together with the problem numbers, should indicate the child-parent relationship between the problems. Because just-in-time type sets can have so many problems you can simplify things by clicking the - icon to hide the children of a particular problem and you can click the + icon to show the children of a particular problem. This is very similar to how viewing a directory structure in list mode works.  You can expand or collapse all of the child problems by using the buttons at the top.

To reorder, you drag a problem by clicking on the arrow icon.  When you move a problem you will be shown a little preview box indicating where the problem will go.  You can change whether a problem is a child or not by dragging the problem slightly to the right or left.  (When you do so the preview box should become indented.)  If you delete a problem there may be gaps in your numbers. This is good if you delete something while a set is active (not recommended) but otherwise you can renumber the problems consecutively using the Renumber button.

Another big difference between this and the original Problem Set Detail page is how problems are rendered.   In this editor you click the picture icon near the problem number to render a problem. If you click it again it will hide the problem. The method used to render the problem is determined by the drop down box near the top.  If you want to render all of the problems at the same time you can use the button at the top of the list. Be warned this will take up a lot of vertical space.  Put together, on the fly rendering and the expand collapse feature should make it so that you can focus on one review sequence at a time.  You can read more about how to set up Just-In-Time sets here.  Below I will present some of the parameters associated to Just-In-Time sets.  

Set Options

These are the set wide options specific to this type of homework. Many of the other set wide options, such as due date, open date and answer date, will be present. If conditional release and/or reduced credit is enabled then these configuration options will also be present.

Restrict Problem Progression 

This option controls whether students are allowed to proceed to the next parent problem before finishing the current problem, including the current problem’s children if necessary. Note: A student "finishes" a problem if they either get 100% or runs out of attempts.

Email Notification 

When this option is enabled all instructors set to receive feedback emails will be sent a notification when a student finishes a problem, e.g. runs out of attempts on a problem, and all child problems which count for that problem's score, and do not have an adjusted status of 100%. This is to let the instructor know that the review for that problem was unsuccessful.

Problem Options

These are options for individual problems. The usual problem variables, such as weight and max attempts, will be present as well.

Att. to Open Children 

This controls how many incorrect attempts are needed before the child problems for this problem become visible to a student. If this is set to 0 then child problems are always visible. If this is set to -1 or a number higher than max attempts then the child problems will become visible after a student runs out of attempts. If this is set to -1 and max attempts is set to unlimited then the child problems will never be open.

Counts For Parent 

This controls whether this problem's score counts towards the score of its parent. In general the adjusted status of a problem is the larger of its status and the weighted average of the status of its child problems with this flag enabled.

While it may not be as fancy as an adaptive system which chooses new problems based on a computer algorithm, this just-in-time assignment type is extremely flexible.  Instructors should be able to make use of the tree structure, as well as the fact that they can control when and how students have access to the homework problems, to do some interesting things.  I know my students would appreciate a short tutorial on the log rules next time we do logarithmic differentiation!