20050501
4 Number of Digits and Precision
k
Number of Digits
Standard Mode
The following applies when the check box next to the “Decimal Calculation” item on the
Basic Format dialog box is not selected.
• Up to 611 digits are stored in memory for integer values.
• Decimal values up to 15 digits are converted to fraction format and saved in memory. When
a mathematical expression cannot be converted to fraction format, the result is displayed in
decimal format.
• Values stored in memory are displayed as-is, regardless of how [Display] settings
(Normal 1, Normal 2, Fix 0 – 9, Sci 0 – 9) are configured (except when a decimal value is
displayed).
Decimal Mode
The following applies when the check box next to the “Decimal Calculation” item on the
Basic Format dialog box is selected.
• Values stored in Ans memory have the same number of digits as they have when
displayed. A value that is assigned to a variable has the same number of digits as defined
for Standard mode values.
• Values are displayed in accordance with how [Display] settings (Normal 1, Normal 2,
Fix 0 – 9, Sci 0 – 9) are configured.
• Displayed values are rounded to the appropriate number of decimal places.
• Some applications store values using a mantissa up to 15 digits long and a 3-digit
exponent.
k
Precision
• Internal calculations are performed using 15 digits.
• The error for a single mathematical expression (Decimal mode calculation error) is
±
1 at
the 10th digit. In the case of exponential display, calculation error is
±
1 at the least
significant digit. Note that performing consecutive calculations causes error to be
cumulative. Error is also cumulative for internal consecutive calculations performed for:
^(
x
y
),
x
,
x
!,
n
P
r
,
n
C
r
, etc.
• Error is cumulative and tends to be larger in the vicinity of a function’s singular point(s) and
inflection point(s), and the vicinity of zero. With sinh(
x
) and tanh(
x
), for example, the
inflection point occurs when
x
= 0. In this vicinity, error is cumulative and precision is poor.
α
-4-1
Number of Digits and Precision
Summary of Contents for ClassPad 300 PLUS
Page 11: ...20050501 ClassPad 300 PLUS ClassPad OS Version 2 20 ...
Page 197: ...20050501 2 10 7 Using Verify 4 Input x 2 1 and press E 5 Input x i x i and press E ...
Page 649: ...20050501 u Graph Line Clustered D u Graph Line Stacked F 13 8 2 Graphing ...
Page 650: ...20050501 u Graph Line 100 Stacked G u Graph Column Clustered H 13 8 3 Graphing ...
Page 651: ...20050501 u Graph Column Stacked J u Graph Column 100 Stacked K 13 8 4 Graphing ...
Page 652: ...20050501 u Graph Bar Clustered L u Graph Bar Stacked 13 8 5 Graphing ...