C 11: Instructions to subjects and data collection

06.773GPPMinimum Performance Requirements for Noise Suppresser Application to the AMR Speech EncoderTS

The instructions given to the subjects will to some extent depend on the method used to collect opinion data. In this section, example instructions are given for Pair Comparison, Modified ACR and CCR experiments. To ensure consistency, the actual instructions given to the subjects should be as close as possible to these examples, adapted for the number of speech files and length of the actual experiment, data collection method, and translated into the native language.

The instructions must be given prior to the commencement of the experiment, and the experimenter should ensure that the subject has understood them before starting the experiment. Questions asked by the subjects on procedural aspects of the experiment can, and should, be answered. However questions about the experiment design or what the experiment is investigating should not be answered until the subjects have completed the experiment. Subjects must be told not to give such information to subjects who are yet to participate in the experiment.

Subjects’ responses may be collected by any convenient method: e.g. pencil and paper, press buttons controlling lamps recorded by the operator, or automatic data-logging equipment. Whichever method is used, care must be taken that subjects should not be able to observe other subjects’ responses, nor should they be able to see the record of their own responses made in a previous session. Apart from the inevitable memory effects, each response should be independent of every other.

C11.1 Example Instructions for Experiment 1

In this test we are evaluating systems that might be used for a type of communications between separate places under a variety of conditions. You are going to hear a number of samples of speech reproduced in the earpieces of the handset. Each sample will consist of a sentence that was produced with two different communication systems. The first is identified as A and the second is identified as B.

Please listen to both A and B and then decide which of the two you prefer. Preference is strictly your decision and the decision should be based on your opinion of the quality of the speech samples. Some of the A/B pairs will seem clearly different and your decisions will be effortless. Others may be more difficult. ALWAYS MAKE A DECISION BETWEEN THE TWO. “I DON’T KNOW” and “I don’t like either one” ARE NOT OPTIONS. Make your decisions independently. You should always compare the two current sentences, and not use any other presentation. Nor, should you be rating whether you like one talker better than another. This is not a test of you in any way; it is an evaluation of the systems. There is no right or wrong. Do not discuss how you are making your ratings during breaks or stretching periods.

For indicating your opinion you are requested to use the button box at your test station. <Use a prototype box to demonstrate with during training>. After listening to the two sentences, all lights on the box will flash. At that time, please press the appropriate single button that represents your opinion of the communication quality of the sample just heard. Use the leftmost button if you preferred the first sample (or A). Use the button on the far right if you preferred the second speech sample (or B). The corresponding light will be activated when a choice has been indicated. Once the button has been pushed, you will not be allowed to change your mind, so please respond carefully.

After you have given your opinion there will be a short pause before the next sample begins.

For practice, you will hear a series and provide an opinion on each; then there will be a break to make sure that everything is clear. An administrator will be in the room to answer questions. From then on you will have a break after each test block (approximately xx minutes). After the test block there will be a three -minute break during which you may leave the room. This series will continue for the duration of the test.

It is imperative that you do a good job with each rating by giving a true opinion of the communication system samples. The ratings you make later in the day are as important as those made earlier in the day. Please stay alert and do you best each time you make a decision.

Thank you for participating in this research. Feel free to ask any procedural questions at this time

C11.2 Example Modified ACR Instructions for Experiment 2

Instructions to the listeners

In this experiment we evaluate systems that might be used for telecommunication service between separate places.

You will hear speech samples reproduced in a telephone handset. Every sample consists of four short unconnected sentences in an environment with varying amounts of background noise. Your task is to indicate your opinion of the overall sound quality with respect to any unnatural sounds leading to unpleasant effects in the sample. Please make your judgement of the sample considering unnatural sound during the complete sample.

Use the following 5-point scale:


no unpleasant effects


slightly unpleasant effects


somewhat unpleasant effects


very unpleasant effects


severely unpleasant effects

After each stimuli there will be a short pause for you to give your opinion. As a practice, you will first hear several samples and give an opinion on each. Then we will check that everything is clear before we start the test. Don’t hesitate to ask questions if you have any. The experiment is divided in four parts with breaks in between. The parts last approximately 15 minutes each. Please do not discuss your opinions with the other participants in the experiment.

Thank you for your participation.

C11.3 Example Instructions for Experiment 3 and 4


In this experiment we are evaluating systems that might be used for telecommunication services.

You are going to hear through the handset pairs of speech samples, most of which have been recorded in different noisy environments (for example inside a car, in an office, or on the street). The first sample you will hear will be the reference sample. You will then hear the same sample again, but this time it will have passed through a telecommunications system. These samples will each consist of two short unconnected sentences.

You should listen carefully to each pair of samples. When they have finished, please record your opinion of the second sample with regard to the first one using the following scale:

Much better


Slightly better

About the same

Slightly worse


Much worse

For practice, you will first hear [n] sample pairs and give an opinion on each. There will then be a short break to make sure that everything is clear.

From then on you will have a break approximately every [p] minutes. The test will last a total of approximately [q] minutes.

Please do not discuss your opinions with other listeners participating in the experiment.