6240B DC Voltage Current Source/Monitor Operation Manual
9.5.3 How to Calculate Accuracy for Low-Resistance Measurement
9-17
Example) Calculating the accuracy in current source of 1 A (1 A range) with a resistance measured value
of 10 m
and in the voltage limiter range 300 mV
1. Reading item calculation
Resistance measurement reading item = Current source setting item + Voltage measurement reading item
= 0.05 % + 0.02 % =
0.07 %
Setting item in the current source 1 A range:
0.05 %
Reading item in the voltage measurement 300 mV: 0.02 %
2. Full-scale item calculation
Full-sale item = (A + Rm
B)/Is
= (10
V + 10 m
300
A)/1 A =
13
From *2,
A: 10
V
(Current limiter range 300 mV)
Rm: 10 m
(Resistance measured value)
B: 300
A
(The current source resolution in the 1A range is 100
A. Thus,
3 digits
100
A = 300
A)
Is: 1 A
(Current source setting value)
From the above results,
Accuracy
=
0.07 %
13
=
(10 m
0.07 %)
13
=
7
13
=
20
Thus, the accuracy is specified between
9.980 m
to 10.020 m
.
