MW1008P, user manual
29
PRECISION
The measurement precision depends on the base precision of the instrument, which itself
depends on the values of impedance and test frequency. This value is increased by a factor
corresponding to the additional error added when impedance is near the min or max values of
its range. The precision values are valid with the use of cables MW-KELV or MW-SMT,
when an open/short calibration has been performed before the measurement, and when the
instrument is in auto ranging mode.
Precision of resistances and impedances
Precision of R
A
r
= ±[A
e
+ (K
H
+K
L
) × 100]%
A
e
:
Base precision, value given in the following table. For Q values > 0,1 multiply
the value of A by (1+|Q|)
K
H
, K
L
:
factors corresponding to range limits error
K
L
= 1 mΩ / Rm
K
H
= Rm / 1 GΩ
K
H
is negligible for resistances smaller than 50 kΩ
K
L
is negligible for resistances greater than 20 Ω
Precision of Q
Q
e
= (A
r
/ 100) (1+Q
x
2
)
With A
r
precision of the resistance calculated above
Calculation example :
Calculation of a resistance measurement precision of 1 kΩ at 1kHz. The instrument gives a
value of Q = Q
x
= 0,0005
From the graphics, you find a base precision of 0,2% = A
e
K
L
= 1 mΩ / 1 kΩ = 10
-6
K
H
= 1 kΩ / 1GΩ = 10
-6
Precision of the resistance = ±(0,2 + 0,0002)% ≈ ± 0,2%
Precision of Q :
∆Q = (0,2/100) ×(1+0,0005
2
) = 0,002
