Section 2
TM9340-G-00
15 August, 2019
2-5
Accuracy
–
Accuracy is based on the most significant dial (MSD) that is used for the resistance
output. The accuracy of the MSD dial used for the resistance output determines the accuracy of
all the dial settings for the resistance output desired. Accuracy for
1 M
Ω
and below is based on subtraction of Zero Resistance (nulling out all dial zero resistances
and lead resistance).
MSD
(M
OST
S
IGNIFICANT
D
IAL
)
U
SED
Accuracy
10 m to 1M 10M & 100M
1G
10G
100G
1T
0.01% + 2
m
Ω
0.1%
1%
2%
5%
6%
Examples show a 9346 Series models dials set to various outputs to show
accuracy calculations.
Example 1
–
A 9346/10k Decade Standard is set to
7
.96912 k
Ω.
The most significant dial would
be 1k. Since the 1k Dial (set to 7 Position) is most significant dial used, the accuracy would be
0.01% + 2
mΩ for all the 9340 Dial Settings
for the entire resistance output.
Example 2
–
A 9346/100M Decade Standard is set to
2
3.1573 M
Ω.
The most significant dial
would be the 10 M
Ω
Dial. For this output, this dial would be set to the 2 Position with remaining
dials set to values as shown. Since the 10
Ω
M dial is the most significant dial used, the accuracy
would be 0.1% for all the 9340 Dial Settings for the entire resistance output.
Example 3
–
A 9346/1T Decade Standard is set to
100
.59
GΩ
. Since the 1 T
Ω
Dial was not used
this resistance requires only the 100 G
Ω
dials and below. The 100 G
Ω
is the most significant dial
used so the accuracy would be 5% for all the 9340 Dial Settings for the entire resistance output. If
the 1 T
Ω
Dial were used as the most significant dial, the accuracy would change to 6% for the
output needed.
