3.15.2 Multi-tenant Spark Thrift Server on Urika-GX
Urika-GX enables tenants to have a tenant-isolated Hive MetaStore, in addition to the Spark Thrift Server. The
metastore allows schema to be shared between
spark-sql
shells and Spark Thrift clients. The Spark Thrift
Server is an 'on demand' service, meaning that users launch it as required and stop it when it is not needed. The
tenant metastore is a pod comprised of two containers running within the tenant's namespace. One of these
containers runs the Hive metastore service, while the other runs a Postgres database.
NOTE:
The metastore allows schema to be shared between
spark-sql
shells and Spark Thrift clients
Hive Metastore Management
Tenant specific metastores are created as part of the tenant creation process. Tenants can use the following
commands within their VM to interact with the metastore:
●
check-metastore
- Checks the status of the metastore, indicates to the user whether or not it is running
and the IP address of the metastore.
●
start-metastore
- Starts the tenant metastore, if there is not one already running.
●
stop-metastore
- Stops the tenant metastore service.
CAUTION: The metastore must be running in order for the Thrift Server to launch successfully.
Spark Thrift Server Management
Tenants can use the following commands within their VM to interact with the Spark Thrift Server:
●
start-thriftserver
- Configures the tenant specific Spark Thrift Server and launches the Spark Thrift
Server spark job in the tenant's Kubernetes namespace
●
stop-thriftserver
- Stops the tenant specific Spark Thrift Server job and does requisite cleanup.
The Spark Thrift Server is a shared service within a tenant, therefore only one Spark Thrift Server instance per
tenant should be running.
Limitations
The current version of Spark on Urika-GX lacks support for long running spark jobs with secure HDFS. As a
result, a Spark Thrift Server instance can only run as long as the HDFS delegation tokens are valid for. Currently
this period is 1 day. After this time, the Spark Thrift Server instance can no longer query data in secure HDFS. If
this happens, the Spark Thrift Server instance must be restarted using the stop/start commands above.
Authentication/Authorization
Tenant users authenticate to the Spark Thrift Server using their LDAP credentials. The Spark Thrift Server relies
on tenant membership for authorizing users. Therefore, if a user is added as a member to a tenant via the
ux-
tenant-add-user
command, the user will be automatically granted access to the Spark Thrift Server. No
separate authorization mechanisms are necessary.
Administering Tenant Spark Thrift Server Components
It may prove useful for a system administrator to be able to check the status of a tenants Spark Thrift Server
and/or metastore. They may do so using kubectl commands from the login node.
To see the state of the Kubernetes POD containing the Spark Thrift Server driver:
System Management
S3016
63
