Run a verification script... (AskiaTools)

This feature in AskiaTools allows you to check the quality of your data by running a custom AskiaScript on your entire data set, or a subset of your data set. For example, you might use a verification script to check that your collected data contains the right data types (such as ensuring that numeric variables contain numeric data), or to confirm that the routing in your survey is working as intended (by checking whether data is present in unexpected combinations of variables). Whenever any condition in your verification script is not met in a particular data record (for example, when data is not found in a mandatory question), AskiaTools will log a warning message.

You can use verification scripts to perform data quality testing in the following situations:

• prior to fieldwork, on randomly generated data;
• during fieldwork, on the first few collected records, or
• after fieldwork, on your final data.

To access this feature first open your qes file, then select run a verification script... in the tools menu.

This topic covers

• Writing a verification script
  • Special keywords used in verification scripts
  • Examples
• Automatically generating verification scripts
• Running a verification script
• Viewing the script results
• Toolbar commands
• Examples

Writing a verification script

Verification scripts consist of a series of conditions, called asserts. If any of these asserts is not satisfied, then a message is logged. For example, you might set up an assert to check that the value in a given numeric variable is above a certain amount. You can define as many asserts as you wish in your script.

Verification scripts use standard AskiaScript syntax, so if you are already familiar with AskiaScript, you will find it easy to write verification scripts. If you are not familiar with AskiaScript, please refer to the AskiaScript reference. Askia also offers a training course on using AskiaScript; please contact Askia if you would like to book a training course.

There are two special script keywords available when defining verification scripts: Assert.Check and GoTo. In addition, while it is not exclusively for use in verification scripts, you may find the Interview object useful, as it enables you to obtain information about the current interview.

An assert takes the following form:

Assert.Check(Q3.HasNoData,"Q3 is empty.")

In this example, if Q3 contains no data in a given data record, the message "Q3 is empty" will be logged for that data record when you run the script.

AskiaTools can automatically generate asserts for you. This feature is commonly used to quickly set up standard tests that confirm that the correct type of data is present in each variable. However, you can define the nature of these generated asserts. If you choose to automatically generate asserts, then AskiaTools will create them for every variable in your QES that does is not already mentioned in an assert statement. Therefore, automatic generation is a good way to supplement any custom asserts you have defined. For information, see automatically generating verification scripts.

It is possible to can save your scripts for future use; a script can be opened and run on any QES. However, if it is to run successfully on a different QES file, the variable names will need to match. For example, if the script references a question called GENDER, then you will need a variable called GENDER present in the current QES. To save a script, simply click save script in the toolbar.

To write your verification script:

1. To begin writing a verification script, you need to open the verification script window in askiatools. First, open your qes file, then select run a verification script... in the tools menu.
2. When you first open the verification script window, you are presented with a blank script, and you can immediately begin entering your script. However, if there is already a script present in the window (i.e. if you have entered a script or opened an existing one), click new script in the toolbar to clear the window and begin a new script:
   
3. Enter the assert statement that make up your script. For detailed information, see assert.
4. To verify your script's syntax, click compile script:
   
   
   
   AskiaTools will report if it finds any incorrect syntax. Note: this syntax check does not guarantee that your script will have the intended effect, as it cannot check the completeness or logic of your script, merely that your script correctly obeys the rules of AskiaScript syntax, and hence will run.

See also:

• Automatically generating a verification script
• Running your script

Special keywords for use in verification scripts

All syntax used in verification scripts is in accordance with AskiaScript 2.0. For more information about askiascript, please refer to the Design Assistant.

There are two special script keywords available when defining verification scripts: Assert.Check and GoTo. The Interview object, while not exclusively for use in verification scripts, is useful when writing them, as it enables you to obtain information about the current interview.

Assert.Check(condition, message)

The Assert object is used to validate the integrity of the interview record's data. Assert.Check evaluates the expression as a boolean value. If the value of the expression is true, nothing happens. Otherwise the assert returns a false value and the specified message is displayed.

 You should use this keyword as follows:

Assert.Check(expression,message) where expression is a variant and message is a string.

For example:
If Country has {1} Then
     Assert.Check(Language has {1},"When country is 'UK', 'English' should be selected, but was "+ ^Language^.Answers[1].Caption)
EndIf

Assert.ToString() returns the string representation of the object/variable.

GoTo (with a corresponding label, which is a name followed by a colon)

The GoTo statement allows you to skip sections of your script.

For example:
Goto Mark1
If Country has {1} Then
     Assert.Check(Language has {1},"When country is 'UK', 'English' should be selected, but was "+ ^Language^.Answers[1].Caption)
EndIf
Mark1:

In this example, the If Country... block  is skipped because the Goto Mark1 script dictates that everything is skipped until it finds Mark1:.

Please note that GoTo can only go forward. It is not possible to use it to go back (upwards) in the script.

The Interview object

The Interview object can be used to obtain information about the current interview. You can use it in the following ways:

• Interview.Broker: returns the Broker ID.
• Interview.BrokerPanelID: returns the broker panel id as received in AskiaWeb (when available).
• Interview.GUID: returns the global unique identifier for each respondent.
• Interview.IPAddress: returns the IP address of the interview (if available).
• Interview.Latitude: returns the latitude for the respondent location (If not available, it will return 0).
• Interview.Longitude: Same as above except it returns longitude.
• Interview.PanelID: returns the Askia Panel ID when available.
• Interview.Progress: returns a percentage value indicating the respondent's progress through the questionnaire.
• Interview.Seed: returns a pseudo-unique number for each interview, used to generate random numbers in the survey.

Question.AllIterations

This keyword will return a value that relates to the number of times a looped question exists in the questionnaire. For example, if Q1 is a question inside a loop with three iterations, then Q1.AllIterations.Count will return a value of 3. If Q2 is a question inside a loop within another loop, with three and two iterations, then Q2.AllIterations.Count will return a value of 6.

Question.HasNoData

This keyword returns a boolean value (true/ false) that indicates whether there is any data in the question.

Question.HasValidData

This keyword indicates if the answer of the current question is valid for open, numeric and closed questions (depending on its type). It checks if there is a value, and if that value is compatible with DK settings (for all question types), the minimum/maximum number of answers and exclusivity (for multi-coded questions), size (for open text questions) and range (for numeric and date questions).

Examples

Suppose Q1 is a multi-coded question inside a loop, and you want to check that it has valid data on all iterations. You might use the following verification script:
Dim i
For i = 1 to Q1.AllIterations.Count
   Assert.Check (Q1.AllIterations[i].HasValidData , "Q1 doesn't have valid data")
Next i
 

Suppose Q3 is asked only if Q2=1. In this case, there are two checks you would need to do:

• trigger an error if Q2 is 1 and there is no data in Q3, and
• trigger an error if you have data in Q3 but Q2 does not equal 1

You might use the following script:

if Q2 has {1} Then
   Assert.Check(Q3.HasValidData,"Skip executed when it shouldn't")
else
   Assert.Check(Q3.HasNoData,"Skip not executed")
EndIf

However, because AskiaTools can compare the return of conditions, we can simplify the above to:
Assert.Check((Q2 has {1})=Q3.HasValidData,"Skip error here")

Because the message is raised when the condition is false, this means both our conditions should be simultaneously true or simultaneously false.

Suppose that q1 is an open question and q2 is a closed (single) question, into which we code the answers given at q1. We want to check that q2 was coded correctly. We create a text file in which we separate each category/answer to be coded using the "|" character and each possible variation of that category/answer using ",". You can easily create such a file with the array of strings within Excel or Word. The way in which we refer to this file and run the check between q1 and q2 is as follows:

Dim strOpen
strOpen = ReadTextFile("Local path to your Test.txt file")

Dim arrRows = strOpen.Split( "|")

Dim i
For i=1 to arrRows.Count
   Dim arrCodes = arrRows[i]
   Dim arrSpells = arrCodes.Split(",")
   Dim j
   For j=1 to arrSpells.Count
      If ^q1^.Value.ToLowerCase() = arrSpells[j].ToLowerCase() Then
         Assert.Check( ^q2^.Value Has i,"Data is wrong coded for "+ ^q1^.Value)
      endif
   Next j
Next i

Automatically generating a verification script

AskiaTools can automatically generate a series of asserts for you. By default, this feature will quickly set up standard tests that confirm that the correct type of data is present in each variable. However, you can customise these generated asserts.

When you use this feature, one assert will be automatically created for each variable in your QES that is not already referenced by an assert statement in your script. For example, if your QES contains Q1, Q2, Q3 and Q4, and your script already mentions Q1 and Q3, then scripts would be generated for Q2 and Q4.

The generated asserts will adhere to the syntax set up in the options window (in the tools menu, select options..., then define the script defaults in the generation of verification script section). For more details, see verification script options.

To automatically generate a verification script:

1. To begin writing a verification script, you need to open the verification script window in AskiaTools. First, open your qes file, then select run a verification script... in the tools menu.
2. When you first open the verification script window, you are presented with a blank script, and you can immediately begin entering your script. However, if there is already a script present in the window (i.e. if you have entered a script or opened an existing one), click new script in the toolbar to clear the window and begin a new script:
   
3. Enter any asserts you want to manually define. Note that asserts will not be automatically generated for any variables you mention in your script. For details on manually defining asserts, see writing a verification script.
4. In the toolbar, click generate script for questions not in current script:
   
   One asserts will be generated for each variables not already referenced in your script.

Running your verification script

Once you have written your verification script, you can run it on either your entire data set or a subset of the data. There are various options available when running your script, for example to diagnose a slow script, or to execute the script line by line to examine the outcome in detail, which we will cover below.

To simply run your script, click run/continue script:

By default, this will run your script on all interviews. During the run, you can pause and resume the script execution by clicking the same icon again. The results of the run are displayed in the results section of the verification scripts window.

The following options determine how your script runs:

• Break run on assert: if selected, the script will be interrupted whenever an assert message is displayed (i.e. when an assert check is failed). This allows you to examine the result and take appropriate action; for example, you may want to halt the run at this point. To continue script execution, simply click run/continue again.
• Select...: allows you to run your script on a sub-set of the data records, instead of the entire data set (for example to perform tests when you do not need to wait for a script to run on a very large data set). When you click this command, a dialog opens in which you can define a script that identifies the records on which you want to runt he data verification. For example, the script ??Gender?? has {1} would ensure that the script only checked records for male respondents (if male was coded as 1 in the gender variable).
• The following options are useful if your script is running slowly, and you want to diagnose this:
  • Run on reduced data: if selected, the script will take into consideration only the questions present in the current script, and will ignore the rest of the database; this allows the script to run faster.
  • Speed test: if selected, the duration and percentage fields will be populated in the results section when you run your script. If your script takes a long time to run, you can use the duration and percentage information to identify the parts of the script that are running slowly.

The following buttons give you control over the script execution during the run:

• Reset run (clear asserts, reposition on first interview): Resets the errors found in the current run (but does not reset the script).
• Restart current interview: Restarts the step-by-step run for the current interview.
• Run script step by step: Using this option, you can step through each part of the script and view the results in the results area of the screen as each step of the script executes. See results for more information. Once you have stepped through all parts of the script for a respondent, the script will start from the beginning again for the next respondent. By default, the script will begin stepping through the first data record in your file. However, you can navigate to another record by using the interview navigation buttons in the toolbar:

  • First interview:  Goes to the first interview in the data file or database.

  • Previous interview: Goes to the previous record in the data file or database.

  • Interview position: Double-click here (between previous interview and next interview) to specify a record number you want to go to.

  • Next interview: Goes to the next record in the data file or database.

  • Last interview: Goes to the last record in the data file or database.

See also:

• Viewing the script results

Viewing the script results

The results section displays the outcome of your script run and is split into four views.

By default, the first view contains the following columns, showing, for each part of your script...

• Type: indicates the type of keyword question, syntax, variable, function, method.
• Value: Indicates the type of the keyword within that category.
• Result: indicates the actual value of the token after it was run.
• If speed test is selected,  the first view also displays columns for duration and percentage information.

The second view in the results section displays a list of interviews that have failed your scripted checks (asserts), along with the details.
The third view in the results section displays a summary. For each line in the script, this view displays the line number, number of interviews checked, number of interviews that failed a check (assert) and the associated message for that check (assert) when relevant.

The fourth view is a console window. It lets you type some code and immediately see the result. For example:

q1.Value + { 1 to 3}

Toolbar commands

The following commands are available in the verification script toolbar:

Icon	Command name	Description
	New script	Allows you to create a new verification script.
	Open existing script...	Allows you to open a previously saved script. Note that scripts are saved with the extension .txt, and can be opened in any text editor. However, we recommend that you edit your scripts within the run a verification script dialog, because AskiaTools will automatically verify your script and highlight any obvious problems (such as invalid variable names or syntax errors).
	Save script	Allows you to save your script as a text file. Note that scripts are saved with the extension .txt, and can be opened in any text editor. However, we recommend that you edit your scripts within the run a verification script dialog, because AskiaTools will automatically verify your script and highlight any obvious problems (such as invalid variable names or syntax errors).
	Compile script	Compiles the script; this must be done prior to running it. As part of the compilation process, AskiaTools examines the syntax of the script, and checks that the script corresponds to the variable names in the database. For more information about AskiaScript syntax, please refer to the AskiaScript section of the AskiaDesign Assistant.
	Generate script for questions not in current script	Automatically generates a default script. The script generated will adhere to the syntax you have declared in the options window (in the tools menu, select options..., then define the script defaults in the generation of verification script section). For more details, see verification script options.
	Run/continue script	Runs the script for all interviews. You can interrupt the run by clicking this command again. If break run on assert is selected, then the script will be interrupted when the first assert message is displayed (i.e. the first incorrect record is encountered). To continue, click run/continue again. If run on reduced data is selected, the script will take into consideration only the questions present in the current script, and will ignore the rest of the database; this allows the script to run faster.
	Reset run (clear asserts, reposition on first interview)	Resets the errors found in the current run (but does not reset the script).
	Restart current interview	Restarts the step-by-step run for the current interview.
	Run script step by step	Using this option, you can step through each part of the script and view the results in the results area of the screen. See results for more information. Once you have stepped through all parts of the script for a respondent, the script will start from the beginning again for the next respondent. By default, the script will run step-by-step on the first respondent in your file. However, you can navigate to another respondent/interview, using the interview navigation buttons.
	Break run on assert	If this option is selected, then the script will be interrupted when the first assert message is displayed (i.e. the first incorrect record is encountered). To continue, click run/continue again.
	Run on reduced data	If this option is selected, the script will take into consideration only the questions present in the current script, and will ignore the rest of the database; this allows the script to run faster. Note: All routings are ignored and all links are removed. This will break anything that refers to ignored responses (and more generally rotation or skipped data information).
	First interview	Goes to the first interview in the data file or database.
	Previous interview	Goes to the previous interview in the data file or database.
	Interview position	Double-click here to specify an interview number you want to go to.
	Next interview	Goes to the next interview in the data file or database.
	Last interview	Goes to the last interview in the data file or database.
	Completes only	If selected, the script will only run on data for completed respondents
	Select...	By selecting this option, you can perform data verification on a sub-set of the data records. When you click this command, a dialog opens in which you can define a script that identifies the records on which you want to runt he data verification. For example, the script ??Gender?? has {1} would ensure that the script only checked records for male respondents (if male was coded as 1 in the gender variable).
	Speed test	If this option is selected, the duration and percentage fields will be populated in the results section when you run your script. If your script takes a long time to run, you can use the duration and percentage information to identify the parts of the script that are running slowly.