$ kubectl logs spark-triangles-1520878896538-driver | grep "riangles:"
numTriangles: 10624230
Number of triangles: 3541410
Resource Configuration for Spark Jobs
The following Spark configuration settings may be used to control the amount of resources that Spark will request
from Kubernetes for any job launched using
spark-submit
under the secure service mode, i.e., under
Kubernetes:
Table 12. Spark Configuration Settings and their Default Values
Configuration Setting
Default Value
Notes
spark.executor.instances
5
Number of Spark executor
containers that will be
launched to provide job
execution under
Kubernetes
spark.executor.cores
1
Number of cores
requested per executor
spark.executor.memory
96g
Amount of memory
requested per executor
spark.driver.cores
1
Number of cores
requested for the driver
This should be increased if
a job does a lot of work in
the driver e.g.
aggregations, result
collection
spark.driver.memory
16g
Amount of memory
requested per driver
CAUTION: Please be aware that due to Kubernetes's service scheduling mechanism, there are always
some services running on all the nodes, using fractional CPU cycles. This may block any requests that
attempt to use the maximum number of cores on the system because a small fraction of those cores is
already allocated.
If a job is not showing any progress, find out the current status of the associated Kubernetes pod by
running
kubectl describe pod
pod-name
. If there are insufficient resources to launch a job, the
system will return a message similar to the following at the end of the output:
Warning FailedScheduling 55s (x8 over 1m) default-scheduler 0/16 nodes are available: 1
PodToleratesNodeTaints, 16 Insufficient cpu.
System Management
S3016
62
