Kubernetes Utility Modules

The TerraNexus Web Application includes the following modules used to perform operations on the TerraNexus Kubernetes Infrastructure:

• createsuperuser module

• delete_admin_db module

• load_default_db module

• restartworkers module

• build_sphinx_docs module

• clearCallbackLogs module

These modules function outside of the normal internal operation of TerraNexus and perform tasks that must be driven from the underlying Kubernetes Infrastructure that support the TerraNexus Web Application.

createsuperuser module

Note

Module Details

Create Superuser Utility Module

This module creates a default superuser privilaged TerraNexus account for testing and development purposes.

Created on 19 Oct 2020

@author: Dr. Matthew B.J. Purss

src.createsuperuser.main()

delete_admin_db module

Note

Module Details

Delete Administration Database Utility Module

External Utility to delete the TerraNexus_admin database

This is a “cleanup” utility used to clear out a corrupted database before loading a clean version in from file.

Created: 31 March 2021

@author: Dr. Matthew B.J. Purss

src.delete_admin_db.main()

Main module function.

This module makes use of the “util/dggs_util.py” utility module but is located outside of the utility file structure for simplicity.

load_default_db module

Note

Module Details

Load Default Database Utility Module

This module loads the default databases into a new deployment of TerraNexus.

Databases automatically deployed include:

• TerraNexus Admin - The general system administration database.

• TerraNexus_S_IT9_2d10b58e2088c7e2 - The default TerraNexus Surface DGGS Instance.

• TerraNexus_V_IT9_2d10b58e2088c7e2 - The default TerraNexus Volumetric DGGS Instance.

Created on 28 Oct 2020

@author: Dr. Matthew B.J. Purss

src.load_default_db._check_if_collections_exists(db, tablenames, db_client)

Internal function to check if a given set of “database Collections” exist on the TerraNexus MongoDB Database Engine.

Parameters:

• db (str, required) – TerraNexus MongoDB database name.

• tablenames (array-like, required) – Array/List of table names (i.e. mongodb collections) to search for.

src.load_default_db._check_if_mongodb_exists(db, db_client)

Internal function to check if a given “database” exists on the TerraNexus MongoDB Database Engine.

Parameters:

db (str, required) – TerraNexus MongoDB database name.

src.load_default_db._connect_mongo(host, port, username, password, atlas_flag)

Internal function to open a connection to the mongoDB Database Engine.

Parameters:

• host (str, required) – TerraNexus MongoDB host address.

• port (str, required) – Authorised connection port for the TerraNexus MongoDB Database Engine.

• username (str, required) – Username for authenticated connection to the TerraNexus MongoDB Database Engine.

• password (str, required) – Password for authenticated connection to the TerraNexus MongoDB Database Engine.

src.load_default_db._drop_mongo_table(db='test_db', tablename='test_table', db_client='')

Internal function to drop (or delete) a “database collection” to the TerraNexus MongoDB Database Engine.

Parameters:

• db (str, required) – TerraNexus MongoDB database name.

• tablename (str, required) – TerraNexus MongoDB database Collection (or Table) name.

src.load_default_db._get_db_to_load(backup_file_prefix='TerraNexus_Backup__TerraNexus_admin*', db_type='admin')

Internal function to get the details of the “database collection” to load into the TerraNexus MongoDB Database Engine.

Parameters:

• backup_file_prefix (str, required) – File prefix to search for to find the database file to load.

• db_type (str, required) – The “type” of TerraNexus MongoDB database to load. Options include: * “admin” * “surface_dggs” * “volumetric_dggs”

src.load_default_db._init_mongo_collection(db='test_db', tablename='test_tbl', db_client='', shard_db=True, capped=False, capped_size=10737418240, index_spec=[])

Internal function to initialise a “database collection” on the TerraNexus MongoDB Database Engine.

Parameters:

• db (str, required) – TerraNexus MongoDB database name.

• tablename (str, required) – TerraNexus MongoDB database Collection (or Table) name.

• shard_db (boolean, optional) – Flag to select whether or not to create a “Sharded” database (where the database is distributed across multiple database nodes). [Default = True]

• capped (boolean, optional) – Flag to select whether or not to “cap” (or limit) the size of the database Collection. [Default = False]

• capped_size (int, optional) – The size in bytes of the “capped” database collection (if the “capped” flag is set to True).

• index_spec (array-like, optional) –

  Index specification for the collection. This specifies which data fields will be set as indices for the collection. Each item in the “index_spec” list is a tuple containing the “field name” to index followed by the order of indexing (i.e. “ASCENDING” or “DESCENDING”)

  Example:

  index_spec = [('morton_code','ASCENDING'),
                  ('cell_id','ASCENDING'),
                  ('res','ASCENDING')]
  

src.load_default_db._init_mongo_database(db='test_db', shard_db=True, db_client='')

Internal function to initialise a “database” on the TerraNexus MongoDB Database Engine.

Parameters:

• db (str, required) – TerraNexus MongoDB database name.

• shard_db (boolean, optional) – Flag to select whether or not to create a “Sharded” database (where the database is distributed across multiple database nodes). [Default = True]

src.load_default_db._load_dbname_to_mongo(db_name='', db_to_load={}, db_files=[], shard_flag=True, db_client='')

Internal function to load the specified “database collection” from file to the TerraNexus MongoDB Database Engine.

Parameters:

• db_name (str, required) – TerraNexus MongoDB database name to load.

• db_to_load (dict, required) – TerraNexus MongoDB database Collection (or Table) details to load.

• db_files (array-like, required) – Array/List of database files to upload to the TerraNexus MongoDB Database Engine.

• shard_flag (boolean, optional) – Flag to select whether or not to create a “Sharded” database (where the database is distributed across multiple database nodes). [Default = True]

src.load_default_db._store_df_to_mongo(db='test_db', tablename='test_table', df=[], db_client='')

Internal function to store a “database collection” to the TerraNexus MongoDB Database Engine.

Parameters:

• db (str, required) – TerraNexus MongoDB database name.

• tablename (str, required) – TerraNexus MongoDB database Collection (or Table) name.

• df (pandas.DataFrame, required) – pandas DataFrame object containing the data to be stored in the TerraNexus MongoDB Database Engine for this Collection.

src.load_default_db.main(host='mongo', port='27017', user='<user>', password='<password>', atlas_flag=False, shard_flag=True)

Main function to load all “default” databases into the TerraNexus MongoDB Database Engine.

Parameters:

• host (str, required) – TerraNexus MongoDB host address.

• port (str, required) – Authorised connection port for the TerraNexus MongoDB Database Engine.

• username (str, required) – Username for authenticated connection to the TerraNexus MongoDB Database Engine.

• password (str, required) – Password for authenticated connection to the TerraNexus MongoDB Database Engine.

• shard_flag (boolean, optional) – Flag to select whether or not to create a “Sharded” database (where the database is distributed across multiple database nodes). [Default = True]

restartworkers module

Note

Module Details

Restart TerraNexus Workers Utility Module

This utility module restarts the TerraNexus “Worker” pods on the TerraNexus Kubernetes Cluster.

A cronjob runs every 60 minutes to restart every Celery “Worker” pod. This is necessary to ensure the connections between the “Worker” pods and the message broker (RabbitMQ) are not stale. If a “Worker” pod looses its connection to the message broker the TerraNexus parallel job queue, and ultimately the performance of TerraNexus will suffer (or stall completely if all workers loose their connection).

Created on 31 Dec 2020

@author: Dr. Matthew B.J. Purss

src.restartworkers.main()

build_sphinx_docs module

Note

Module Details

TerraNexus Documentation Builder Utility Module

This Module dynamically generates the TerraNexus documentation. It uses the Sphinx Python Documentation Generator to build the documentation from structured comments throughout the TerraNexus codebase.

This Module is run as part of every build and deployment of the TerraNexus Web Application to ensure the software documentation is always synchronised with the state of the software codebase.

Created on 2 July 2022

@author: Dr. Matthew B.J. Purss

src.build_sphinx_docs.main(terranexus_build, mongodb_host, mongodb_port, mongodb_user, mongodb_password, mongodb_shard_flag, local_docker_build=False, docker_password='')

clearCallbackLogs module

Note

Module Details

Clear Callback Logs Utility Module

This module clears the TerraNexus asynchronous callback logs generated by asynchronous (or subscriber mode) operation of OGC API Processes deployed on the TerraNexus OGC API infrastructure.

This module is run via a cronjob at midnight (UTC) everyday operating on the TerraNexus Kubernetes Cluster.

Created on 17 May 2023

@author: Dr. Matthew B.J. Purss

src.clearCallbackLogs.main()