PROCESSING
SPECIFICATIONS
9
8
1. Valid Number of Digits
The valid numbers of digits for processing are as follows.
The valid number of digits using A to describe the valid number of
digits of the input data (number of digits after the decimal point) is
indicated in the following table.
Valid Number of Digits
Valid Number of Digits
(After Decimal Point)
Symbol Meaning
Tolerance
Data Input
data
A
N
Number of data
0
MAX Max.
value
A
MIN Min.
value
A
R Range
A
X
Average
A + 2
Lowermost digit
1
n
Standard deviation
A + 2
Lowermost digit
1
n-1
Standard deviation
A + 2
Lowermost digit
1
P
Fraction defective
3 (**.***%)
Lowermost digit
1
Cp
Process capability index
3
Lowermost digit
1
Cpk
Process capability index
3
Lowermost digit
1
LSL
Lower limit value
A
Lowermost digit
1
USL
Upper limit value
A
Lowermost digit
1
DIV
Number of divisions in histogram
Fixed to 10
Histogram range display
A + 2
Lowermost digit
1
X
Center (X control)
A + 2
Lowermost digit
1
_
_
X-UCL
Upper control limit (X control)
A + 2
Lowermost digit
1
_
_
X-LCL
Lower control limit (X control)
A + 2
Lowermost digit
1
_
_
R
Center (R control)
A + 2
Lowermost digit
1
_
R-UCL
Upper control limit (R control)
A + 2
Lowermost digit
1
_
R-LCL
Lower control limit (R control)
A + 2
Lowermost digit
1
_
IMPORTANT
For the processing error application range, refer to
Processing Error Details.
- 43 -
www.calcert.com
1.800.544.2843
0
5
10
15
20
25
30