Follow these steps to quickly set up an experiment:
Some functionality of this application is restricted to users who have an account, such as creating an experiment or viewing results. In this documentation, processes that require an account will be denoted with "(A)" in their title. If you believe you require an account but do not have one, please contact us.
Users without an account can participate in experiments.
All experiments require a title and description. These will be presented to the participant when they start the experiment, so provide necessary information about the experiment that the participant will need to know. Note that participants are told how to answer each question. The title of an experiment will also be used by admins to identify an experiment from others when viewing, all experiments, so choose their name carefully.
Images or videos are required for every experiment. When uploading files, they must be named "[PREFIX]_[SUFFIX]", where the prefix is the name of the original file and the suffix is the algorithm applied. Files can be organised in one of two ways:
Files must have consistent naming, as described above, as well as consistent resolutions for sets of images. Either of these options can be modified in the source code by removing the appropriate function calls to parse the data.
Currently, three different types of questions are supported for a given set of files:
Using one of the above you can have participants judge files by some metric, for example simply having the participants judge them by how good they look. Admins have the option to customise these questions, below we will explain each questions options.
Some options exist that apply to all types of questions:
Pairwise Comparison
Ranking
Rating
Admins can also change the colour of the text and background that will be displayed the the participant during the test.
Admins also have the option to include a "refresher image". This will cover the entire display and be shown for a period of time between questions before disappearing. If selected, admins can choose whether to have a specific colour be shown, or a specific image, as well as the time (in milliseconds) that it will be shown for.
Once the admin has entered valid options and clicked the create button, they will be redirected to a link containing information about the experiment, which we will explain in the "Experiment Details (A)" section below.
Admins can view all experiments by clicking the respective link on the home page. Here, they will be presented with a list of all previous experiments, sorted such that the most recently created are at the top. Each experiment is identified by its title, its type of questions (e.g. ranking) and the date and time it was created.
Clicking any of the experiment titles will navigate the admin to a page which allows them to manage such experiment, more is explained in the below "Experiment Details (A)" section.
This page holds details about an experiment such as:
There also exists the option to access an experiments results. Either the admin can download the .csv file, or they can choose to go to another page to view more extensive results in-browser without having to download and query the results. Depending on the experiment type more columns may be added to the .csv file, if so this will be explained to the user. The results are explained more in depth below.
For each experiment we want to know how each participant answered the questions, thus we need to be able to differentiate between several participants answering the experiment, perhaps simultaneously. To manage this we use Django sessions. Each user is "signed up" for an experiment upon starting it, and their session is given a unique key to link that user with that experiment. Thus, on all future HTTP GET requests for experiment questions, the participants client can provide such key to authenticate with the server that they have registered for such experiment. This functionality prevents participants from answering multiple experiments at once. Note, if a participant has already "signed up" for an experiment and re-enters the consent form, they will be registered under a new key, so their results will appear under a different user.
Admins can also choose to delete experiments on this page - a confirmation is shown to help prevent mistakes.
To navigate to this link participants must be given its URL, for convenience this is displayed to the admin after they create the experiment, on the Experiment Details page.
Before a participant can begin an experiment, they are forced to check that they consent to several statements. This can be altered if required in the source code. This page also displays the title of the experiment, as well as its description that was given when creating the experiment.
After a participant has consented to everything and clicked the button, their client will be registered with the experiment (explained more in depth above in "Experiment Results (A)"), and they will be redirected to the first question.
The design and functionality of each question will be very different depending on the options, so we will break this into separate sections for each type of question.
In every type of question, participants can enlarge images to their original resolution by clicking on them. This will darken the display and show the respective image at the center of the screen. Clicking anywhere on the display will remove this view. Note that sometimes images are resized depending on the resolution of the display, but they will never be scaled above their original resolution and the original image can be seen by clicking it.
Participants will be shown several files, and be asked to choose one. They can select a file by clicking the radio choice, or its label, next to the file.
If the user chose the overlay option, the images will be displayed one at a time in the center, with the option to switch between which one is displayed. If not chosen, if there are more than four files, they will be displayed in rows with four files per row. The files may also be scaled depending on the display resolution, as we noted earlier.
Participants will be shown several files in a vertical column, and will be asked to arrange these in some ordering of their preference. They can change the ordering by holding down the handle associated with each file and dragging it up or down.
Note, for ranking participants must click on every file to enlarge them before they can proceed to the next question.
Participants are shown several files and are asked to rate each one. Depending on how the admin set up the experiment, they will have to answer in a particular way:
If there are more than four files, they will be displayed in rows with four files per row. The files may also be scaled depending on the display resolution, as we noted earlier. Participants can also select a files rating when it is enlarged (not for radio though), for convenience.