Table 42. Hadoop Error Messages
Error Message
Description
Resolution
org.apache.hadoop.hdfs.server.\
namenode.SafeModeException:
Cannot create or delete a file. Name
node is in safe mode.
During the start up, the NameNode
goes into a safe mode to check for
under replicated and corrupted
blocks. A Safe mode for the
NameNode is essentially a read-
only mode for the HDFS cluster,
where it does not allow any
modifications to file system or
blocks. Normally, the NameNode
disables the safe mode
automatically, however, if there are
there are too many corrupted
blocks, it may not be able to get out
of the safe mode by itself.
Force the NameNode out of safe
mode by running the following
command as a HDFS user:
$
hdfs dfsadmin -safemode
leave
Too many underreplicated blocks in
the NameNode UI
Couple of dataNodes may be down.
Please check the availability of all
the dataNodes
If all the DataNodes are up and still
there are under replicated blocks.
Run the following 2 commands in
order as a HDFS user:
$
hdfs fsck / | grep 'Under replicated' |
awk -F':' '{print $1}' >> \
/tmp/under_replicated_files
$
for hdfsfile in `cat /tmp/
under_replicated_files`; \
do echo "Fixing $hdfsfile :" ; \
hadoop fs -setrep 3 $hdfsfile; \
done
Too many corrupt blocks in name
node UI
The NameNode might not have
access to at least one replication of
the block.
Check if any of the DataNodes are
down. If all the DataNodes are up
and the files are no longer needed,
execute the following command:
$
hdfs fsck / -delete
org.apache.hadoop.ipc.\
RemoteException(java.io.IOExcepti
on): \ File /tmp/test could only be
replicated to \ 0 nodes instead of
minReplication (=1).
HDFS space may have reached full
capacity. Even though Urika-GX has
a heterogeneous file system, the
default storage type is DISK unless
explicitly set to use SSD.
The user might have filled up the
default storage, which is why HDFS
would not be able to write more
data to DISK.
To identify the used capacity by
storage type, use the following
commands:For both
DISK
and
SSD
,
calculate the sum of usage on all
the DataNodes.
For
DISK
:
$
df /mnt/hdd-2/hdfs/dd | awk
'NR==2{print $3}'
For
SSD
:
$
df /mnt/ssd/hdfs/dd | awk
'NR==2{print $3}'
YARN job is not running. You can
see the status of the job as
ACCEPTED: waiting for AM
container to be allocated, launched
and register with RM.
The NodeManagers may not be
running to launch the containers.
Check the number of available node
managers by executing the
following command:
$
yarn node -list
Troubleshooting
S3016
267
