Mistake in 2005 HSC? (2 Viewers)

[pkc]

Anyone notice a possible mistake in Q12 of multiple choice?

"12) A student performed a sampling technique as part of a first-hand investigation on water quality and repeated the sampling technique several times.
What aspect of the experiment was improved by repeating the procedure?"

The marking notes say the answer is B) Reliability.

The definition of first-hand reliability in BOS Science Syllabus 7-10 states that it is "the degree to which repeated observations and/or measurements taken under identical circumstances will yield the same results."

How can the degree of similarity between results be increased by taking more of them, if anything the spread will get bigger ???

 

[Riviet]

You need to consider the other options in the question, can any of the others be more correct than reliability? Repeating an experiment does not improve the safety, nor does it ensure that your method was valid.
Accuracy may improve to a small extent, if you take out the outlying results, but accuracy of an experiment is best improved by using more precise equipment such as pipette and burette in a titration experiment.
Repeating the experimnt several times does improve your reliability as even if you have particular trials which yield results that exceed the theoretical value or a less than it, you should notice a similarity in a number of your results, e.g you could get several results around the 50 ml mark in a titration experiment and some obvious outlying results, which can be taken out of your averaging and calculations. All those results around 50 ml will help confirm the reliability of your total results. I hope that helps.

 

[pkc]

Fair enough,

However you seem to be applying a different definition to that stated by the BOS for first-hand investigations. You are using the general meaning of "dependability or credibility of the information" in the way it is used to assess 2nd hand sources.

If this definition is applied however then the only way the data's dependability or credibility could be increased is if its accuracy was thought to have increased.

This would make both A and B equally correct.

Also, I do not see how the BOS definition of first hand reliability can be interpreted as meaning simply the number of measurements which are the same, without comparing it to the number of measurements which are not the same. Surely the spread of the data is more the indication of how reliable the data could be.

 

[insert-username]

How can the degree of similarity between results be increased by taking more of them, if anything the spread will get bigger ???

No, if you get more of them and they're very similar or the same, then the degree is increased. If you get wildly different results, then the degree of similarity decreases. In this case, if you do the procedure more times, and you get the same results each time, reliability is increased (from your definition), since the degree to which repeated measurements under identical circumstances yield the same results has increased. Doing the procedure more times does not improve accuracy. Conducting the experiment slower and with better equipment improves accuracy much more than repetition.

Also, I do not see how the BOS definition of first hand reliability can be interpreted as meaning simply the number of measurements which are the same, without comparing it to the number of measurements which are not the same. Surely the spread of the data is more the indication of how reliable the data could be.

You're right - the BOS definition by its very nature includes a comparison to the number of measurements which are not the same. If the degree to which repeated measurements give the same result is small, then a large number of results are spread right out. If the degree is high, the spread is much narrower. Hence, if the results are closely clustered around a certain point, the reliability is good, and if the results are spread right out, the reliability is not so good.

In any case, the overriding theory with multiple choice is to select the most correct answer. In this case, the most correct answer is definitely reliability, not accuracy. They are not equally correct.


I_F

 

[helper]

If the same method is used repeatibly, the spread of the results will form a statistical bell curve. So if you increase your sample size, the mean of your values will be closer to the statistical value. So by increasing your number of samples, you do decrease the spread of values, you are less likely to have statistical outliers.

 

[pkc]

Doing the procedure more times does not improve accuracy.

I_F

Surely the reported result (calculated average) of the experiment should be more likely to be accurate if it is based on more measurements ??

 

[pkc]

Thanks All,
I guess from reading these posts there are many interpretation of reliability and "similarity of results" that we all have and I guess Im really after a reason for why the following argument could not be made:

Suppose a student were to interpret the BOS definition of first hand reliability as:
"the degree to which repeated observations and/or measurements taken under identical circumstances will yield the same results."
as,
"the standard deviation or spread of the results from an experiment if the experiment was carried out a huge number of times."

If this was the case, then no matter how many times the experiment was actually carried out, the reliability could not be changed - it is a fixed value that depends on the equipment, method and operator.

What if a student also suggested that the more times an experiment is done, the more likelihood that the reported result from the experiment (the calculated average) would be more accurate ie close to the true value.

Why wouldn't this be a reasonable argument to support answer A- Accuracy rather than answer B- Reliability?

Cheers

 

[insert-username]

Surely the reported result (calculated average) of the experiment should be more likely to be accurate if it is based on more measurements ??

What if a student also suggested that the more times an experiment is done, the more likelihood that the reported result from the experiment (the calculated average) would be more accurate ie close to the true value.

Why wouldn't this be a reasonable argument to support answer A- Accuracy rather than answer B- Reliability?

Your arguments are reasonable for answer A - accuracy. Once again though, the more correct argument is for reliability, since the Board of Studies' definition for reliability relates directly to reproducibility (i.e. "to the degree to which repeated measurements give the same result), which is what the question is asking. Remember: two answers may be correct, but one is always more correct.

If accuracy was the answer they wanted, they might have asked, "Why did the student use a pipette instead of a measuring cylinder?". Yes, a case can be made for reliability there, but the more correct answer is accuracy, because the Board of Studies associates accuracy with the inherent uncertainties in measurements, e.g. measuring a very small length with a metre ruler. By using a pipette, the student is ensuring that their results are more accurate.


I_F

 

[pkc]

Your arguments are reasonable for answer A - accuracy. Once again though, the more correct argument is for reliability, since the Board of Studies' definition for reliability relates directly to reproducibility (i.e. "to the degree to which repeated measurements give the same result), which is what the question is asking.

Exactly!
Reliability, by the BOS definition is the "reproducibility" - or the ability of the experiment (equipment, method and operator) to produce results which are similar (close or clustered) and can be quantified by measurements of "standard deviation" and "range" done on a very large number of results that come from the particular experiement.

The ability of the experiment to produce a certain amount of spread in a large number of results has nothing to do however with how many times the experimenter happens to choose to carry it out.

 

[pkc]

Here's an example of what I'm getting at:

Lets say Alice wants to analyse the pH of a sample of water with happens to have true pH value of 7.00.

Alice chooses to do the analysis with a pH probe and meter.

Initially Alice gets results of say 6.98,7.01, 7.02 etc.

As she monitors the spread of her data (say by calculating standard deviation) she notices that it changes a lot at first 0.02,0.01,0.03 but starts to settle around 0.02 as more measurements are taken. In fact it gets to the point that no matter how many more times she does the experiment the spread stays at 0.02. The width of the bell curve is now showing up.

The reliability ( "degree to which repeated observations and/or measurements taken under identical circumstances will yield the same results") for Alice and the pH probe has now been determined, sd=0.02, or however you want to express it.

Whether the experiment is now done once, twice or 10 million times, it doesn't matter! The reliability of the method is set and the results will approach this same degree of spread each time (providing equipment, operator and conditions stay the same).

 

[insert-username]

The ability of the experiment to produce a certain amount of spread in a large number of results has nothing to do however with how many times the experimenter happens to choose to carry it out.

Wrong. An experiment cannot produce a certain spread of results unless it is carried out repeatedly. If I do the experiment once, is there a spread? If it is carried out ten times and the close cluster of results obtained, the results are reliable. If it is carried out even more times and you still get a very close spread, it is even more reliable.

Whether the experiment is now done once, twice or 10 million times, it doesn't matter! The reliability of the method is set and the results will approach this same degree of spread each time (providing equipment, operator and conditions stay the same).

From your example, accuracy is set, since accuracy is to do with not only the uncertainties in measurement, but the deviation from the accepted value (i.e. the standard deviation of the results from the correct value). Hence, Alice's results are accurate since they have a very small deviation from the accepted value, 7. However, the more times she does it, the more reliable her results are, as she's repeated the procedure considerably and gotten the same results - hence, other people doing the same procedure can expect to get the same results as her. However, her accuracy does not improve, because, as you say, the standard deviation of all her results has settled at 0.02 (in fact, repetition has caused accuracy to improve and worsen until the deviation settled).

Example: two scientists a few decades ago announced that they had managed to create a cold fusion power source (in short, limitless power from a cup of water, so to speak). However, no one else could do it, and their discovery was discredited. Why was it discredited? Because it couldn't be repeated, and hence their work was not reliable. Would creating the cold fusion source again make their discovery more accurate? Not at all. What does creating the power source again do? Make the experiment more reliable.

Going back again, you don't seem to be getting my main point. The BoS definition for reliability is directly related to reproducibility, and since the question is asking why the procedure is repeated, the most correct answer is reliability.


I_F

 

[pkc]

Yes I agree, its definitely about which answer is more correct. Let me put the argument for answer B as incorrect this way:

If I can run 100m in an average of say 13s (shown by doing the run a large number of times) then that is my ability at the moment - and it wont change by taking a sample of 1 result or 20 results

In the same way the ability of an experiment to produce results which are tightly clustered is set, simply because the width of the bell curve (that fits the graph of a large number of results from the experiment) is set.

To say that reliability (by the BOS definition) increases with sample size is like saying standard deviation decreases by taking more samples.

This is untrue. Standard deviation fluctuates around a certain number (both above and below) and then settles on the number as you take more samples. It's the number that is settled on which is the measure of spread or reliability for the experiment. It is the spread of the entire population of results that could have come from the experiment, or the "degree to which repeated observations and/or measurements taken under identical circumstances will yield the same results").

However the accuracy of the reported result (the average) is always increasing as you gather more samples. It is "zeroing-in" on the true value.
This would make answer A the more correct answer.

 

[Riviet]

However the accuracy of the reported result (the average) is always increasing as you gather more samples. It is "zeroing-in" on the true value.
This would make answer A the more correct answer.

But... it is not the accuracy that increases. You are still using the same equipment to conduct the experiment and record results, therefore you have not improved your accuracy, but averaging all your results will yield more reliable results.

 

[lala2]

Accuracy is how close the results are to the 'bull's eye'. Reliability is the 'spread' of data. By repeating, you are minimising the chance of outliers that could affect the 'spread' of your data. That does not mean it's accurate, because if your equipment is calibrated wrongly, then all results would be inaccurate, i.e. nowhere close to the 'bull's eye'. However, repetition would mean that you're consistently getting reliably wrong results (but you wouldn't know that they're wrong unless you compared your results with someon else's)

 

[pkc]

But... it is not the accuracy that increases. You are still using the same equipment to conduct the experiment and record results, therefore you have not improved your accuracy, but averaging all your results will yield more reliable results.

This would be true if accuracy was defned by the equipment used, but it is not.

Accuracy is defined as the closeness of a result to the true-value. In this case the reported average of the water analysis would be likely to be closer to the true-value because of a larger number of samples taken.

Of course changing equipment can effect accuracy as well, but it's certainly not the only way to effect it.

 

[pkc]

Accuracy is how close the results are to the 'bull's eye'. Reliability is the 'spread' of data. By repeating, you are minimising the chance of outliers that could affect the 'spread' of your data. That does not mean it's accurate, because if your equipment is calibrated wrongly, then all results would be inaccurate, i.e. nowhere close to the 'bull's eye'. However, repetition would mean that you're consistently getting reliably wrong results (but you wouldn't know that they're wrong unless you compared your results with someon else's)

Good point!
An improvement in accuracy of the reported result is not guaranteed for the reasons you point out (calibration maybe out etc).

However it does make an improvement in accuracy more likely (given that you dont know whether the calibration is faulty or not).

This is what makes Accuracy the more correct answer compared to Reliability which has no chance of being correct when compared to the BOS definition.

 

[insert-username]

Good point!
An improvement in accuracy of the reported result is not guaranteed for the reasons you point out (calibration maybe out etc).

However it does make an improvement in accuracy more likely (given that you dont know whether the calibration is faulty or not).

This is what makes Accuracy the more correct answer compared to Reliability which has no chance of being correct when compared to the BOS definition.

I really don't get what you mean here, and your arguments don't connect together. If your equipment is calibrated wrong, repeating is not going to make it more accurate - you'll just get more inaccurate results. Recalibrating the equipment will improve your accuracy. Repetition won't, since you're using the same equipment, operator, and method, like you said earlier. What repetition will do is make the results more reliable. They may be completely wrong, but they'll be reliable results that can be shown to be wrong through the inaccurate calibration.

Anyways, the question asks about repeating the experiment. The BoS definition of reliability directly references repeatability, whereas accuracy is related to the closeness to the "correct" value and the measuring errors in the experiment. Hence, the most correct answer is reliabilty.


I_F

 

[pkc]

What repetition will do is make the results more reliable.

I_F

As stated before, repetition does nothing to reduce the spread of data.

As you take more samples the standard deviation of the samples fluctuates both up and down around a horizontal midline.

If repetition decreased the spread in the data, the standard deviation amongst the samples would go down continuously, or at least follow a downward trend, as more samples were taken. This is not the case.

 

[NightShadow]

I'm going to come out and say that repeating an experiment also increases the validity of an experiment, to reduce the fanciful occurrences of freak experimental results... and by doing that ... it improves the accuracy of the data values - which equals to reliability... accuracy relates to the actual experiment.. you have not changed anything... at all... all you've done... is introduced a larger set of values... which improves the accuracy of the data set... which makes them more reliable - conforming with the BOS definition

and wher might you say.. is evidence that introducing a larger set of values will improve the accuracy of a data set? meh... natural phenomena... larger data sets allow for more outrageous deviations to be accounted for... go do some standard deviation calculations:

basically variance is standard deviation squared
and variance has a denominator of [number of variables - 1]

look..i'm not going to explain it.. fact is.. larger data sets... generally equal smaller standard deviations due to the accountabilty of experimental error

 

[pkc]

That's an interesting take on things nightshadow!

which improves the accuracy of the data set... which makes them more reliable - conforming with the BOS definition

However the BOS wouldn't like your definition of reliability (for first hand data)

"the degree to which repeated observations and/or measurements taken under identical circumstances will yield the same results."

-its only to do with spread of data, -nothing to do with the accuracy of the point they centre around