●
Grafana and InfluxDB logs - Grafana logs are stored under
var/log/grafana
, whereas InfluxDB logs are
stored under
var/log/influxdb
. Influxdb log files are compressed. Both Grafana and InfluxDB use the
logrotate
utility to keep log files from using too much space. Log files are rolled daily by default, but if
space is critical, logs can be deleted manually.
●
Spark logs - Shuffle data files on the SSDs is automatically deleted on Urika-GX. Spark logs need to be
deleted manually and are located at the following locations:
○
Spark event logs - Located at
hdfs://user/spark/applicationHistory
○
Spark executor logs - Located on individual compute nodes at
/var/log/mesos/agent/slaves/
●
Hadoop logs - Hadoop log files are located in the following locations and need to be deleted manually:
○
Core Hadoop - Log files are generated under the following locations:
▪
var/log/hadoop/hdfs
▪
var/log/hadoop/yarn
▪
var/log/hadoop/mapreduce
○
ZooKeeper - ZooKeeper logs are generated under
var/log/zookeeper
○
Hive (metastore and hive server2) - These logs are generated under
var/log/hive
○
Hive Webhcat - These logs are generated under
var/log/webhcat
○
Oozie - Oozie logs are stored under
/var/log/oozie
○
HUE - HUE logs are generated under
/var/log/hue
●
Flex scripts (urika-yam-status, urika-yam-flexup, urika-yam-flexdown, urika-yam-flexdown-all) - These
scripts generate log files under
/var/log/urika-yam.log
and need to be deleted manually.
●
mrun
-
mrun
does not generate logs.
●
Cray Graph Engine (CGE) logs - The path where CGE log files are located is specified via the
-l
parameter
of the
cge-launch
command. Use the
cge-cli log-reconfigure
command to change the location after
CGE is started with
cge-launch
. CGE logs need to be deleted manually. Users can also use
--log-level
argument to CGE CLI commands to set the log level on a per request basis. In addition, the
cge.server.DefaultLogLevel
parameter in the
cge.properties
file can be used to set the log level to
the desired default.
8.7
Diagnose and Troubleshoot Orphaned Mesos Tasks
Prerequisites
This procedure requires root access and the system to be running in the default service mode. Execute the
urika-state
command to ensure that Mesos is running and that the system is operating in the default service
mode.
About this task
The metrics displayed in Mesos UI can also be retrieved using CURL calls. Cray-developed scripts (for flexing up
a YARN sub-cluster) and
mrun
use these curl calls in as they interoperate with Mesos for resource brokering. If
Troubleshooting
S3016
258
