TerraNexus Utilities Package

TerraNexus Utilities Submodules

The following submodules are defined by this package.

• util.dggs_operations module

• util.dggs_util module

• util.logging_middleware module

• util.ui_tools module

• util.slack_logger module

util.dggs_operations module

Note

Module Details

DGGS Operations Utility Script

This Module contains the collection of operations and parallel job functions to support the operation of both Surface and Volumetric DGGS Instances registered to the TerraNexus Web Application Infrastructure.

These functions need to be able to be called outside of the Class Object structure defined in util.dggs_util module.

This module, coupled with the surface_dggs.scripts.surface_dggs_setup module, the volumetric_dggs.scripts.volumetric_dggs_setup module as well as the common utility functions defined in the util.dggs_util module, represents a production grade implementation of Patent WO2018/136996 Purss and Pangaea-Innovations-Pty.-Ltd. [2017]

Created: 9 February 2018

@author: Dr. Matthew B.J. Purss

class src.util.dggs_operations.RobustMongoDBBatchTracker(mongo_host='mongo', mongo_port='27017', mongo_user=None, mongo_pwd=None, db_name='batch_tracking', connection_timeout=5000, server_selection_timeout=5000, max_pool_size=100)

Bases: object

Robust batch tracking with MongoDB and retry logic

_connect()

Establish MongoDB connection with retry logic

_ensure_indexes()

Create necessary indexes with retry logic

_get_collection()

Get collection with automatic reconnection if needed

add_error_log(batch_id, error_message)

Add an error log entry to a batch

add_retry_history(batch_id, retry_info)

Add retry history to a batch

close()

Close MongoDB connection

create_batch(batch_number, total_batches, total_tasks, metadata=None)

Create a new batch with retry logic

get_batch(batch_id)

Get complete batch document

get_batch_status(batch_id)

Get current batch status with retry logic

get_incomplete_batches(older_than_minutes=60)

Find batches that are stuck or incomplete

mark_batch_failed(batch_id, reason='Unknown')

Mark a batch as failed

reset_task_status(batch_id, file_index)

Reset a task’s status for retry

update_batch_field(batch_id, field, value)

Update a specific field in a batch document

update_batch_status(batch_id, status)

Update the status of a batch

update_task_completion(batch_id, result=None, error=None)

Update batch when a task completes or fails with retry logic

src.util.dggs_operations._update_callback(task_utils, callback_json, progress, message)

Helper function to update and publish callback

src.util.dggs_operations._update_provenance_activity(task_utils, job_id, provChain)

Helper function to update provenance activity

src.util.dggs_operations.batch_process_data_files(job_json_gridffs, data_file_list, job_df, raw_data_db)

Process data files in batches for better memory management

src.util.dggs_operations.calculate_polygon_coverage(task_utils, task_params, job_df, dggs, featureTypeID, spatioTemporalGeometry_df, res, num_intervals, spatialOnlyFlag, max_feature_size, rec_data_df, rec_properties_df)

Calculate polygon coverage for tesseract DGGS

src.util.dggs_operations.check_batch_status(batch_id)

Check the status of a specific batch

src.util.dggs_operations.check_job_status(job_result)

Check the status of all batches in a job

src.util.dggs_operations.cleanup_completed_batches(days_old=7)

Clean up old completed batches

src.util.dggs_operations.cleanup_references(job_df, dggs, data_type, dataset_id, job_json_gridffs, data_file)

Clean up references to help with memory management

src.util.dggs_operations.clearCallbackLogs()

Note

Function Details

Function to clear the OGC Callback Logs from the database. This is to help manage the size of the Callback logs over time.

This function is called by the clearCallbackLogs module Utility Module every midnight (UTC time) and will purge all records older than 24 hours old.

src.util.dggs_operations.create_collection_entity(dataset_id, job_id, job_df, datafilename)

Create collection entity for provenance

src.util.dggs_operations.create_dggs_instance(job_df, dggs_params)

Create appropriate DGGS instance based on job parameters

src.util.dggs_operations.create_map_execution_agent()

Create the map execution agent info

src.util.dggs_operations.create_process_data_agent()

Create the process data agent info

src.util.dggs_operations.create_reorganise_activity(task_params, job_id)

Create activity for reorganise task

src.util.dggs_operations.create_reorganise_agent()

Create agent for reorganise task

src.util.dggs_operations.create_surface_dggs_tasks(task_params, job_df, dggs, featureTypeID, df_chunks, zonesToRefine, res)

Create tasks for surface DGGS processing

src.util.dggs_operations.create_tesseract_geometry_df(dggs, rec_data_df, rec_properties_df, job_df, task_params)

Create geometry dataframe for Tesseract DGGS

src.util.dggs_operations.create_test_points(radial_points_df, job_df, rec_properties_df, spatialOnlyFlag, dggs, spatioTemporalGeometry_df)

Create test points for polygon sampling

src.util.dggs_operations.create_volumetric_dggs_tasks(task_params, job_df, dggs, featureTypeID, df_chunks, zonesToRefine, res)

Create tasks for volumetric DGGS processing

src.util.dggs_operations.fetch_additional_data(job_json_gridffs, data_json, job_df, raw_data_db)

Fetch additional data if paginated

src.util.dggs_operations.fetch_next_page(url, retries=3, timeout=30)

Fetch next page with retries and timeouts

src.util.dggs_operations.find_cooincident_data()

[NOT IMPLEMENTED YET]

src.util.dggs_operations.force_batch_completion(batch_id, mongo_tracker)

Force a batch to complete when all tasks are processed

src.util.dggs_operations.generate_radial_points(feature_bbox, feature_bbox_centroid, radial_range, theta_range, spatioTemporalGeometry_df, task_utils)

Generate radial points to sample polygon interior

src.util.dggs_operations.generate_zones(job_id='----')

Note

Function Details

data import function to take an existing terrain dataset (X,Y,Z or Lon,Lat,height) and register it to voxels within the volumetric DGGS

Parameters:

job_id (str, optional, default='-') – The unique identifier of the data mapping job.

Returns:

job_output – JSON compatible dictionary object is returned with Job/Process Outputs

Return type:

JSON

src.util.dggs_operations.get_FMSDI_TideData(get_latest_data=False, get_timeseries=False, data_channels=[], startTs=0, endTs=0)

Note

Function Details

Internal Function to get Tidal Data available under the OGC FMSDI 2025 pilot

src.util.dggs_operations.get_child_zone_coordinates_callback(task_id, task_result)

Note

Function Details

callback function that recieves and manages the output from the get_child_zone_coordinates_task asynchronous Celery Task.

Parameters:

• task_id (str, required) – identifier of the task currently being run.

• task_result (JSON, required) – task results in JSON format

src.util.dggs_operations.get_child_zone_coverage_callback(task_id, task_result)

Note

Function Details

callback function that recieves and manages the output from the get_child_zone_coverage_task asynchronous Celery Task.

Parameters:

• task_id (str, required) – identifier of the task currently being run.

• task_result (JSON, required) – task results in JSON format

src.util.dggs_operations.get_chunk_size(job_df)

Get chunk size for processing

src.util.dggs_operations.get_feature_data(task_utils, task_params, featureTypeID)

Get feature data from MongoDB

src.util.dggs_operations.get_feature_parameters(job_df, rec_properties_df, dggs, baseGeometry_ll)

Get feature parameters for mapping (height, time, etc.)

src.util.dggs_operations.get_job_data(task_utils, job_id)

Get job data from database

src.util.dggs_operations.get_mapped_child_zones_callback(task_id, task_result)

Note

Function Details

callback function that recieves and manages the output from the get_child_zone_coordinates_task asynchronous Celery Task.

Parameters:

• task_id (str, required) – identifier of the task currently being run.

• task_result (JSON, required) – task results in JSON format

src.util.dggs_operations.get_mapped_data(dggs_store, dggs_specs, mapped_data_files, search_type='All_data', ref_location=[], search_box=[], search_radius=1.0, res_min=0, res_max=1, mesh_params={}, output_type='data_points', output_format='json', chunk_size=100)

Note

Function Details

Internal Function to retrieve mapped data from the database

Parameters:

• dggs_specs (dictionary) –

  dictionary of initialisation parameters to instantiate the volumetric DGGS. For example

  {'resolution_max':      15,
   'refinement_ratio':    9,
   'max_radius_multiple': 2,
   'shell_dggs_type':    'TerraNexus',
   'base_polyhedron':    'cube',
   'cell_type':          'square',
   'earth_model':        'GRS80',
   'a':                  6378137.0,
   'f':                  1.0,
   'ref_lon':            0.0,
   'ref_lat':            0.0}
  

• mapped_data_files (str) – terrain dataset to import into the volumetric DGGS

• search_type (str) –

  type of search to perform valid options are

  * All = get all registered cells
  * cellID = get all registed cells for the given list of cellIDs
  * res = get all registered cells for the given range of resolutions
  * search_box = use a search box to perform a Morton_code range search
  * range_search = use a distance search from a reference location
  

• ref_location (array-like, optional) – list or numpy array with the [lon, lat, height] of the reference location for range search [Default = []]

• search_box (dictionary, optional) – a dictionary containing the coordinates for the search box vertices [Default = {}]

• search_radius (float, optional) – search radius (in metres) from a reference location to perform a radial search on [Default = 0]

• res_min (int, optional) – Minimum DGGS resolution to store the terrain in [Default = 0]

• res_max (int, optional) – Maximum DGGS resolution to store the terrain in [Default = 15]

• mesh_params (JSON, optional, default={}) – The data meshing parameters to send to PyMesh. If no mesh_params are provided a set of hard coded mesh parameters will be used.

• output_type (str, optional, default="data_points") – Output data type.

• output_format (str, optional, default="json") – Output data format.

• chunk_size (int, optional, default=100) – Chunk size to use to throttle the process to ensure it does not exceed the hardware memory resource capacity.

Returns:

execution_output – A JSON/Dictionary object providing information about the mapped data and status of the mapping job.

Return type:

JSON

src.util.dggs_operations.get_new_zones_callback(task_id, task_result)

Note

Function Details

callback function that recieves and manages the output from the get_new_zones_task asynchronous Celery Task.

Parameters:

• task_id (str, required) – identifier of the task currently being run.

• task_result (JSON, required) – task results in JSON format

src.util.dggs_operations.get_voxels_callback(task_id, task_result)

Note

Function Details

callback function that recieves and manages the output from the get_voxels_task asynchronous Celery Task.

Parameters:

• task_id (str, required) – identifier of the task currently being run.

• task_result (JSON, required) – task results in JSON format

src.util.dggs_operations.get_zoneTile_callback(task_id, task_result)

Note

Function Details

callback function that recieves and manages the output from the get_zoneTile_task asynchronous Celery Task.

Parameters:

• task_id (str, required) – identifier of the task currently being run.

• task_result (JSON, required) – task results in JSON format

src.util.dggs_operations.handle_exception(e, task_utils, function_name)

Handle and report exceptions

src.util.dggs_operations.handle_retry_exception(self, e, task_utils, function_name)

Handle exceptions with retry logic

src.util.dggs_operations.health_check()

Check MongoDB connection health

src.util.dggs_operations.init_footprint_collection(dggs, task_params, job_df, job_id, featureTypeID, provChain, mongoShardFlag, generatedList)

Initialize collection for DGGS zone footprint

src.util.dggs_operations.init_properties_collection(task_utils, dataset_id, job_id, job_df, datafilename, provChain, generatedList)

Initialize collection for feature properties

src.util.dggs_operations.initialise_registered_zones(dggs_ID, searchBox, res_max=1, equalise_cells_flag=False)

Note

Function Details

function to build out the registered zones for a given search box. This will increase efficiency of DGGS query and mapping operations.

Parameters:

• dggs_ID (str, required) – The DGGS Instance Identifier.

• searchBox (JSON, required) – a dictionary/JSON object containing the coordinates for the search box vertices [Default = {}]

• res_max (int, optional, default=1) – The maximum DGGS Resolution Level.

• equalise_cells_flag (boolean, optional, default=False) – A Flag to selected whether or not to Equalise the DGGS Zone Area.

src.util.dggs_operations.initialize_dataset_collections(dggs, dataset_id)

Initialize MongoDB collections for dataset storage

src.util.dggs_operations.interpolate_path(path_to_segment, num_intervals)

Note

Function Details

Utility function to pad out a path with linear interpolated points between the initial path vertices.

Parameters:

• path_to_segment (array_like, required) – the path (defined by an array of vertices) to pad and interpolate between.

• num_intervals (int, required) – number of intervals to interpolate over

Returns:

output_path – output path with the new interpolated vertices

Return type:

array_like

src.util.dggs_operations.interpolate_zoneCoverage_path(coords_array, num_intervals)

Interpolate points along a path to ensure proper zone coverage

src.util.dggs_operations.map_data(job_id='----', job_df={}, provChain={}, callback_json={})

Data import function to take an existing terrain dataset (X,Y,Z or Lon,Lat,height) and register it to voxels within the volumetric DGGS.

Parameters:

• job_id (str, optional, default='-') – The unique identifier of the data mapping job.

• provChain (Dictionary-Like, optional) – Provenance Chain information necessary to log Provenance.

• callback_json (JSON, optional) – Callback message (in JSON format) to send to the Process Callback Listeners.

Returns:

job_output – JSON compatible dictionary object with Job/Process Outputs

Return type:

JSON

src.util.dggs_operations.map_data_execution(job_id, provChain={}, callback_json={})

Internal function to execute the data mapping operation.

Parameters:

• job_id (str, required) – The unique identifier of the data mapping job.

• provChain (Dictionary-Like, optional, default = {}) – Provenance Chain information for logging.

• callback_json (JSON, optional, default = {}) – Callback message to send to Process Callback Listeners.

Returns:

execution_output – JSON/Dictionary with info about mapped data and mapping job status.

Return type:

JSON

src.util.dggs_operations.map_feature_geometry(task_utils, geom_json, feature_json, feature_id, dataset_id, job_id, job_df, dggs, dggs_dimensionality, provChain)

Map feature geometry to DGGS based on geometry type

src.util.dggs_operations.map_voxels_callback(task_id, task_result)

Note

Function Details

callback function that recieves and manages the output from the map_voxels_task asynchronous Celery Task.

Parameters:

• task_id (str, required) – identifier of the task currently being run.

• task_result (JSON, required) – task results in JSON format

src.util.dggs_operations.map_zones_to_features_callback(task_id, task_result)

Note

Function Details

callback function that recieves and manages the output from the map_zones_to_features_task asynchronous Celery Task.

Parameters:

• task_id (str, required) – identifier of the task currently being run.

• task_result (JSON, required) – task results in JSON format

src.util.dggs_operations.mesh_cells(dggs_specs, mesh_params, search_area_cells, chunk_size=10000)

Note

Function Details Function to create 3D Meshes of Volumentric DGGS Zones

Parameters:

• dggs_specs (JSON, required) –

  JSON representation of DGGS instance specification. For example

  {'resolution_max':      15,
   'refinement_ratio':    9,
   'max_radius_multiple': 2,
   'shell_dggs_type':    'TerraNexus',
   'base_polyhedron':    'cube',
   'cell_type':          'square',
   'earth_model':        'GRS80',
   'a':                  6378137.0,
   'f':                  1.0,
   'ref_lon':            0.0,
   'ref_lat':            0.0}
  

• mesh_params (JSON, optional, default={}) – The data meshing parameters to send to PyMesh. If no mesh_params are provided a set of hard coded mesh parameters will be used.

• search_area_cells (array-like, required) – The array/list of search area Surface DGGS ZoneId’s

• chunk_size (int, optional, default=10000) – Chunk size to use to throttle the process to ensure it does not exceed the hardware memory resource capacity.

src.util.dggs_operations.mesh_voxel(dggs_specs, mesh_params, voxel_to_mesh, chunk_size=10000)

Note

Function Details Function to create 3D Meshes of Volumentric DGGS Zones

Parameters:

• dggs_specs (JSON, required) –

  JSON representation of DGGS instance specification. For example

  {'resolution_max':      15,
   'refinement_ratio':    9,
   'max_radius_multiple': 2,
   'shell_dggs_type':    'TerraNexus',
   'base_polyhedron':    'cube',
   'cell_type':          'square',
   'earth_model':        'GRS80',
   'a':                  6378137.0,
   'f':                  1.0,
   'ref_lon':            0.0,
   'ref_lat':            0.0}
  

• mesh_params (JSON, optional, default={}) – The data meshing parameters to send to PyMesh. If no mesh_params are provided a set of hard coded mesh parameters will be used.

• voxel_to_mesh (str, required) – The DGGS ZoneID of Voxel to mesh

• chunk_size (int, optional, default=10000) – Chunk size to use to throttle the process to ensure it does not exceed the hardware memory resource capacity.

src.util.dggs_operations.mesh_voxels_callback(task_id, task_result)

Note

Function Details

callback function that recieves and manages the output from the mesh_voxels_task asynchronous Celery Task.

Parameters:

• task_id (str, required) – identifier of the task currently being run.

• task_result (JSON, required) – task results in JSON format

src.util.dggs_operations.orthometric_to_ellipsoidal(verticalDatum, heightMin, heightMax, baseGeometry_ll, task_params={})

src.util.dggs_operations.parse_temporal_value(value)

Parse temporal value to datetime

src.util.dggs_operations.process_batch_with_robust_mongodb(batch_tasks, batch_number, total_batches, metadata=None)

Process a batch of tasks using MongoDB with retry logic

src.util.dggs_operations.process_chunked_data(task_utils, task_params, job_df, dggs, featureTypeID, rec_data_df, zonesToRefine, res, chunkSize)

Process feature data in chunks

src.util.dggs_operations.process_complex_tesseract_feature(task_utils, task_params, job_df, dggs, featureTypeID, rec_data_df, rec_properties_df, spatioTemporalGeometry_df, res, num_intervals, spatialOnlyFlag, max_feature_size)

Process a feature that needs interpolation for Tesseract DGGS

src.util.dggs_operations.process_data_file(task_utils, job_json_gridffs, data_file, dataset_id, job_df, dggs, dggs_dimensionality, job_id, datafilename, data_type, provChain, callback_json, generatedList)

Process a data file and map it to DGGS

src.util.dggs_operations.process_data_file_batch(job_json_gridffs, batch_files, job_df, raw_data_db)

Process a batch of data files

src.util.dggs_operations.process_data_files_in_batches(job_id, data_file_list, provChain, callback_json)

Process data files in batches using MongoDB-backed tracking

src.util.dggs_operations.process_feature(task_utils, feature_json, feature_index, dataset_id, job_id, job_df, dggs, dggs_dimensionality, datafilename, provChain, generatedList)

Process an individual feature

src.util.dggs_operations.process_feature_collection(job_json_gridffs, data_json, job_df, data_file, raw_data_db)

Process a feature collection

src.util.dggs_operations.process_feature_data(task_utils, task_params, collectionName, a0, f)

Process feature data for reorganization

src.util.dggs_operations.process_feature_properties(task_utils, feature_json, feature_id, dataset_id)

Process and store feature properties

src.util.dggs_operations.process_feature_zones(task_utils, task_params, job_df, dggs, featureTypeID, rec_data_df, baseGeometry_ll, heightMin, heightMax, temporalMin, temporalMax, temporal_flag, spatialOnlyFlag, mongoShardFlag, generatedList)

Process feature to identify and map to DGGS zones

src.util.dggs_operations.process_features_in_batches(task_utils, feature_list, dataset_id, job_id, job_df, dggs, dggs_dimensionality, datafilename, provChain, callback_json, generatedList)

Process features in batches to manage memory usage

src.util.dggs_operations.process_geometry_collection(geom_json, feature_json, feature_id, dataset_id, job_id, job_df, dggs, provChain, chunkSize)

Process geometry collection - handle each geometry type separately

src.util.dggs_operations.process_geometry_collection_linestring(geom, feature_json, feature_id, dataset_id, job_id, job_df, dggs, provChain, chunkSize)

Process linestring geometry in a geometry collection

src.util.dggs_operations.process_geometry_collection_multilinestring(geom, feature_json, feature_id, dataset_id, job_id, job_df, dggs, provChain, chunkSize)

Process multilinestring geometry in a geometry collection

src.util.dggs_operations.process_geometry_collection_multipoint(geom, feature_json, feature_id, dataset_id, job_id, job_df, dggs, provChain, chunkSize)

Process multipoint geometry in a geometry collection

src.util.dggs_operations.process_geometry_collection_multipolygon(geom, feature_json, feature_id, dataset_id, job_id, job_df, dggs, provChain, chunkSize)

Process multipolygon geometry in a geometry collection

src.util.dggs_operations.process_geometry_collection_point(geom, feature_json, feature_id, dataset_id, job_id, job_df, dggs, provChain, chunkSize)

Process point geometry in a geometry collection

src.util.dggs_operations.process_geometry_collection_polygon(geom, feature_json, feature_id, dataset_id, job_id, job_df, dggs, provChain, chunkSize)

Process polygon geometry in a geometry collection

src.util.dggs_operations.process_higher_resolution(task_utils, task_params, job_df, dggs, featureTypeID, rec_data_df, res, chunkSize, mongoShardFlag)

Process higher resolution DGGS zones

src.util.dggs_operations.process_level0_zone(task_utils, task_params, job_df, dggs, featureTypeID, rec_data_df, zoneID, mongoShardFlag)

Process a level 0 DGGS zone

src.util.dggs_operations.process_linestring(geom_json, feature_json, feature_id, dataset_id, job_id, job_df, dggs, provChain, chunkSize)

Process linestring geometry by dividing into manageable chunks

src.util.dggs_operations.process_multilinestring(geom_json, feature_json, feature_id, dataset_id, job_id, job_df, dggs, provChain)

Process multilinestring geometry - split into individual linestrings

src.util.dggs_operations.process_multipoint(geom_json, feature_json, feature_id, dataset_id, job_id, job_df, dggs, provChain, chunkSize)

Process multipoint geometry - split into individual points

src.util.dggs_operations.process_multipolygon(geom_json, feature_json, feature_id, dataset_id, job_id, job_df, dggs, provChain, chunkSize)

Process multipolygon geometry - split into individual polygons

src.util.dggs_operations.process_point_chunk(task_utils, task_params, job_df, dggs, featureTypeID, chunk, rec_data_df, res, spatialOnlyFlag)

Process a chunk of sample points for polygon coverage

src.util.dggs_operations.process_sample_points_in_batches(task_utils, task_params, job_df, dggs, featureTypeID, test_points_df, rec_data_df, res, spatialOnlyFlag)

Process sample points in batches for memory efficiency

src.util.dggs_operations.process_simple_tesseract_feature(task_utils, task_params, job_df, dggs, featureTypeID, spatioTemporalGeometry_df, res, spatialOnlyFlag, lowerGeometry_ll=[], upperGeometry_ll=[], baseGeometry_ll=[])

Process a feature that doesn’t need interpolation for Tesseract DGGS

src.util.dggs_operations.process_spatial_dggs(task_utils, task_params, job_df, dggs, featureTypeID, rec_data_df, rec_properties_df, mongoShardFlag, generatedList)

Process feature mapping for Surface or Volumetric DGGS

src.util.dggs_operations.process_tesseract_dggs(task_utils, task_params, job_df, dggs, featureTypeID, rec_data_df, rec_properties_df)

Process feature mapping for Tesseract (Spatio-Temporal) DGGS

src.util.dggs_operations.process_tesseract_resolution(task_utils, task_params, job_df, dggs, featureTypeID, rec_data_df, rec_properties_df, spatioTemporalGeometry_df, res, spatialOnlyFlag, lowerGeometry_ll=[], upperGeometry_ll=[], baseGeometry_ll=[])

Process feature mapping for a specific Tesseract DGGS resolution

src.util.dggs_operations.process_volumetric_geometry(task_utils, task_params, dggs, featureTypeID, rec_data_df, dggs_zone, feature_in_zone, zoneID)

Process geometry types for volumetric DGGS

src.util.dggs_operations.publish_message_to_group(message: Dict[str, Any], group: str) → None

Note

Function Details

Internal Function to publish messages to a Django Channels Web Socket Connection Group.

Parameters:

• message (JSON, required) – The message to publish to the Web Socket Connection Group

• group (str, required) – The Django Channels Web Socket Connection Group to publish the message to.

src.util.dggs_operations.recover_stuck_batch(batch, mongo_tracker)

Recover a stuck batch by reprocessing failed or missing tasks

src.util.dggs_operations.recreate_dggs_zone(job_df, dggs, zoneID, dggs_params)

Recreate DGGS zone if not found

src.util.dggs_operations.reprocess_missing_tasks(batch, mongo_tracker)

Identify and reprocess tasks that were never started

src.util.dggs_operations.restart_all_selected_pods(namespace, podname_prefix='rabbitmq', force_restart=False)

Note

Function Details

Internal function to get the list of RabbitMQ “worker” pods to restart and to call the process to perform the restart.

src.util.dggs_operations.restart_kubernetes_pod(namespace, pod)

Note

Function Details

Internal function to perform the restart the specified RabbitMQ “worker” pods.

src.util.dggs_operations.restart_pod_callback(task_id, task_result)

Note

Function Details

callback function that recieves and manages the output from the restart_pod_task asynchronous Celery Task.

• this forces a refresh of the connection between worker and broker

Parameters:

• task_id (str, required) – identifier of the task currently being run.

• task_result (JSON, required) – task results in JSON format

src.util.dggs_operations.retry_failed_tasks(batch, mongo_tracker)

Retry tasks that previously failed

src.util.dggs_operations.retry_mongodb_operation(operation_func, max_retries=5, logger=None)

Execute a MongoDB operation with retry logic for connection issues

Parameters:

• operation_func (callable) – Function that performs the MongoDB operation. Should take no arguments.

• max_retries (int, optional) – Maximum number of retry attempts, default 5

• logger (logging.Logger, optional) – Logger object to record retry attempts and errors

Returns:

The result of the operation_func if successful

Return type:

Any

Raises:

Exception – The last exception encountered after max_retries

src.util.dggs_operations.retry_with_exponential_backoff(max_retries=5, base_delay=0.5, max_delay=30)

Decorator for retrying MongoDB operations with exponential backoff

src.util.dggs_operations.rotated_mesh_callback(task_id, task_result)

Note

Function Details

callback function that recieves and manages the output from the rotated_mesh_task asynchronous Celery Task.

Parameters:

• task_id (str, required) – identifier of the task currently being run.

• task_result (JSON, required) – task results in JSON format

src.util.dggs_operations.setup_dggs(task_utils, job_df)

Setup DGGS based on job parameters

src.util.dggs_operations.setup_provenance(task_utils, task_params, job_id, featureTypeID)

Setup provenance agents and activity

src.util.dggs_operations.setup_surface_dggs(dggs_params)

Set up a Surface DGGS

src.util.dggs_operations.setup_tesseract_dggs(dggs_params)

Set up a Tesseract (Spatio-Temporal 4D) DGGS

src.util.dggs_operations.setup_volumetric_dggs(dggs_params)

Set up a Volumetric DGGS

src.util.dggs_operations.store_feature_geometry(task_utils, feature_json, feature_id, dataset_id)

Store feature geometry in GridFS

src.util.dggs_operations.store_tesseract_coverage(task_utils, task_params, job_df, featureTypeID, path_zoneIds, res)

Store tesseract zone coverage information

src.util.dggs_operations.store_tesseract_zone_mappings(task_utils, task_params, job_df, featureTypeID, res_zoneIds, path_zoneIds, res, spatioTemporalGeometry_df={}, lowerGeometry_ll=[], upperGeometry_ll=[], baseGeometry_ll=[])

Store tesseract zone mappings

src.util.dggs_operations.store_zone_coverage(task_utils, task_params, job_df, featureTypeID, zone_in_feature, zoneID)

Store zone coverage information

src.util.dggs_operations.update_feature_bounding_box(task_utils, task_params, rec_data_df)

Update feature properties with bounding box information

src.util.dggs_operations.update_map_execution_activity(task_utils, job_id, provChain)

Update the map execution activity

src.util.dggs_operations.update_process_data_activity(task_utils, job_id, provChain)

Update the process data activity

src.util.dggs_operations.update_provenance_activity(task_utils, task_params, job_id, featureTypeID, generatedList)

Update provenance activity with completion time and generated entities

src.util.dggs_operations.update_reorganise_activity_completion(task_utils, task_params, provActivity)

Update provenance activity with completion time

util.dggs_operations tasks

src.util.dggs_operations.get_surrounding_voxels_task(dggs_specs={}, res=[0], search_region={}, height_min=6378136.0, height_max=6378138.0, search_radius=6378137.0)

Note

Function Details

Celery Task to search for registered voxels for a given resolution within a search radius from the vertices of a given search region in parallel.

Parameters:

• dggs_specs (JSON, required) –

  JSON representation of DGGS instance specification. For example

  {'resolution_max':      15,
   'refinement_ratio':    9,
   'max_radius_multiple': 2,
   'shell_dggs_type':    'TerraNexus',
   'base_polyhedron':    'cube',
   'cell_type':          'square',
   'earth_model':        'GRS80',
   'a':                  6378137.0,
   'f':                  1.0,
   'ref_lon':            0.0,
   'ref_lat':            0.0}
  

• res (arry-like, required) – The list of resolutions to search for surrounding Voxels

• search_region (JSON, required) – The search_region defined as a JSON object.

• height_min (float, optional, default=6378136.0) – The minimum ellipsoidal height from the centre of the Earth in metres.

• height_max (float, optional, default=6378137.0) – The maximum ellipsoidal height from the centre of the Earth in metres.

• search_radius (float, optional, default=6378137.0) – The search radius in metres.

Returns:

output_dict – JSON compatible dictionary object is returned with details of the tested DGGS Zone.

Return type:

JSON

src.util.dggs_operations.process_features_task(task_params={})

Note

Function Details

Celery Task to Process Feature data and map it to a specified DGGS instance in parallel.

Parameters:

task_params (JSON, required) –

JSON representation of Task Parameters.

{'parent_cell_id' : "A123",
'searchBox': [[1231.0,-2132342.0,0.0],
              [-3334234.2342, 5534343.000, -123123.0],
              [2234234.00, 12312333.00334, 234233.05],
              [44345234, -12312333.0,550535.0],
              [1231.0,-2132342.0,0.0]]
'res' : 5,
'dggs_ID' : TerraNexus_Tesseract_e065a3f5b27a8795,
'equalise_cells_flag' : True}

Returns:

output_dict – JSON compatible dictionary object is returned with details of the tested DGGS Zone.

Return type:

JSON

src.util.dggs_operations.map_zones_to_features_task(mapped_data_db='mapped_data_db', selected_feature='feature_id', feature_type_id='', level_0_zone='', job_id='')

Note

Function Details

Celery Task to Map DGGS Zones to Features in parallel.

Parameters:

• mapped_data_db (str, required) – The MongoDB database containing the Feature data to be mapped

• selected_feature (str, required) – The Selected Feature to map to the DGGS Instance.

• level_0_zone (str, required) – The level 0 DGGS Zone to begin the generation of new DGGS Zones from.

• job_id (str, required) – The unique identifier for the processing job.

Returns:

output_dict – JSON compatible dictionary object is returned with details of the tested DGGS Zone.

Return type:

JSON

src.util.dggs_operations.map_zoneTiles_to_features_task(mapped_data_db='mapped_data_db', selected_feature='feature_id', feature_type_id='', zoneID='', job_id='', dggs_ID='', res_min=0, res_max=1, equalise_cells_flag=False)

Note

Function Details

Celery Task to Map DGGS Zone Tiles to Features in parallel.

Parameters:

• mapped_data_db (str, required) – The MongoDB database containing the Feature data to be mapped

• selected_feature (str, required) – The Selected Feature to map to the DGGS Instance.

• zoneID (str, required) – The DGGS Zone to begin the generation of new DGGS Zone Tiles from.

• dggs_ID (str, required) – The DGGS Instance Identifier.

• res_min (int, optional, default=0) – The minimum DGGS Resolution Level.

• res_max (int, optional, default=1) – The maximum DGGS Resolution Level.

• equalise_cells_flag (boolean, optional, default=False) – A Flag to selected whether or not to Equalise the DGGS Zone Area.

Returns:

output_dict – JSON compatible dictionary object is returned with details of the tested DGGS Zone.

Return type:

JSON

src.util.dggs_operations.map_zoneCoordinates_to_features_task(mapped_data_db='mapped_data_db', selected_feature='feature_id', zoneID='', job_id='', dggs_ID='', res_min=0, res_max=1, equalise_cells_flag=False)

Note

Function Details

Celery Task to Map DGGS Zone Coordinates to Features in parallel.

Parameters:

• mapped_data_db (str, required) – The MongoDB database containing the Feature data to be mapped

• selected_feature (str, required) – The Selected Feature to map to the DGGS Instance.

• zoneID (str, required) – The DGGS Zone to begin the generation of new DGGS Zone Tiles from.

• dggs_ID (str, required) – The DGGS Instance Identifier.

• res_min (int, optional, default=0) – The minimum DGGS Resolution Level.

• res_max (int, optional, default=1) – The maximum DGGS Resolution Level.

• equalise_cells_flag (boolean, optional, default=False) – A Flag to selected whether or not to Equalise the DGGS Zone Area.

Returns:

output_dict – JSON compatible dictionary object is returned with details of the tested DGGS Zone.

Return type:

JSON

src.util.dggs_operations.map_zoneCoverages_to_features_task(mapped_data_db='mapped_data_db', selected_feature='feature_id', zoneID='', job_id='', dggs_ID='', res_min=0, res_max=1, equalise_cells_flag=False, pyramid_coverage_flag=True)

Note

Function Details

Celery Task to Map DGGS Zone Coverages to Features in parallel.

Parameters:

• mapped_data_db (str, required) – The MongoDB database containing the Feature data to be mapped

• selected_feature (str, required) – The Selected Feature to map to the DGGS Instance.

• zoneID (str, required) – The DGGS Zone to begin the generation of new DGGS Zone Tiles from.

• dggs_ID (str, required) – The DGGS Instance Identifier.

• res_min (int, optional, default=0) – The minimum DGGS Resolution Level.

• res_max (int, optional, default=1) – The maximum DGGS Resolution Level.

• equalise_cells_flag (boolean, optional, default=False) – A Flag to selected whether or not to Equalise the DGGS Zone Area.

Returns:

output_dict – JSON compatible dictionary object is returned with details of the tested DGGS Zone.

Return type:

JSON

src.util.dggs_operations.map_points_to_voxels_task(mapped_data_file='', mapped_voxels_file='', chunk_id='', selected_data=[], dggs_specs={}, search_area_voxels=[])

Note

Function Details

Celery Task to Map Points to Volumetric DGGS Zones in parallel.

Parameters:

• mapped_data_file (str, required) – The file containing the Feature data to be mapped

• mapped_voxels_file (str, requried) – The file containing the DGGS Voxels

• chunk_id (str, required) – The identifier for the chunk of data to map.

• selected_data (array-like, required) – The Selected data to map to the DGGS Instance.

• dggs_specs (JSON, required) –

  JSON representation of DGGS instance specification. For example

  {'resolution_max':      15,
   'refinement_ratio':    9,
   'max_radius_multiple': 2,
   'shell_dggs_type':    'TerraNexus',
   'base_polyhedron':    'cube',
   'cell_type':          'square',
   'earth_model':        'GRS80',
   'a':                  6378137.0,
   'f':                  1.0,
   'ref_lon':            0.0,
   'ref_lat':            0.0}
  

• search_area_voxels ([], required) – List of search area voxels.

Returns:

output_dict – JSON compatible dictionary object is returned with details of the tested DGGS Zone.

Return type:

JSON

src.util.dggs_operations.mesh_data_task(data_points, mesh_params={}, data_mesh_file='')

Note

Function Details

Celery Task to create 3D Meshes of point data in parallel.

Parameters:

• data_points (array-like, required) – The data points to be meshed

• mesh_params (JSON, optional, default={}) – The data meshing parameters to send to PyMesh. If no mesh_params are provided a set of hard coded mesh parameters will be used.

• data_mesh_file (str, required) – The filename of the data mesh file to save.

Returns:

output_dict – JSON compatible dictionary object is returned with details of the tested DGGS Zone.

Return type:

JSON

src.util.dggs_operations.mesh_voxel_task(voxel_id, dggs_specs={}, mesh_params={}, equalise_zone_flag=False)

Note

Function Details

Celery Task to create 3D Meshes of Volumentric DGGS Zones in parallel.

Parameters:

• voxel_id (str, required) – The Volumetric DGGS ZoneId

• dggs_specs (JSON, required) –

  JSON representation of DGGS instance specification. For example

  {'resolution_max':      15,
   'refinement_ratio':    9,
   'max_radius_multiple': 2,
   'shell_dggs_type':    'TerraNexus',
   'base_polyhedron':    'cube',
   'cell_type':          'square',
   'earth_model':        'GRS80',
   'a':                  6378137.0,
   'f':                  1.0,
   'ref_lon':            0.0,
   'ref_lat':            0.0}
  

• mesh_params (JSON, optional, default={}) – The data meshing parameters to send to PyMesh. If no mesh_params are provided a set of hard coded mesh parameters will be used.

Returns:

output_dict – JSON compatible dictionary object is returned with details of the tested DGGS Zone.

Return type:

JSON

src.util.dggs_operations.mesh_cell_task(cell_id, dggs_specs={}, mesh_params={})

Note

Function Details

Celery Task to create 3D Meshes of Surface DGGS Zones in parallel.

Parameters:

• cell_id (str, required) – The Surface DGGS ZoneId

• dggs_specs (JSON, required) –

  JSON representation of DGGS instance specification. For example

  {'resolution_max':      15,
   'refinement_ratio':    9,
   'max_radius_multiple': 2,
   'shell_dggs_type':    'TerraNexus',
   'base_polyhedron':    'cube',
   'cell_type':          'square',
   'earth_model':        'GRS80',
   'a':                  6378137.0,
   'f':                  1.0,
   'ref_lon':            0.0,
   'ref_lat':            0.0}
  

• mesh_params (JSON, optional, default={}) – The data meshing parameters to send to PyMesh. If no mesh_params are provided a set of hard coded mesh parameters will be used.

Returns:

output_dict – JSON compatible dictionary object is returned with details of the tested DGGS Zone.

Return type:

JSON

src.util.dggs_operations.celery_task_loader(task_type, data_type='model', dggs_specs={}, mesh_params={}, search_region={}, search_radius=6378137.0, height_min=6378136.0, height_max=6378138.0, resolution_range=[], search_area_voxels=[], search_area_cells=[], data_points=[], mapped_data_file='', mapped_data_db='', mapped_data_ids=[], rotation_axis=[], rotation_angle=0.0, ref_coord_xyz=[], mesh_df=[], chunk_size=10000, pyramid_coverage_flag=True)

Note

Function Details

Celery Task that acts as a task loader to sub-tasks in parallel.

Parameters:

• task_type (str, required) – The name of the task to load and run.

• dggs_specs (JSON, optional, default={}) –

  JSON/Dictionary object with the specifications for the selected Surface DGGS Instance. For example

  {'resolution_max':      15,
   'refinement_ratio':    9,
   'max_radius_multiple': 2,
   'shell_dggs_type':    'TerraNexus',
   'base_polyhedron':    'cube',
   'cell_type':          'square',
   'earth_model':        'GRS80',
   'a':                  6378137.0,
   'f':                  1.0,
   'ref_lon':            0.0,
   'ref_lat':            0.0}
  

• parent_cells (array-like, optional, default=[]) – The array/list of parent zones to refine.

• search_radius (float, required) – The Search Radius in metres from a Reference Location (i.e. the centroid of the search box)

• search_box_xyz (array-like, required) – The Search Box in Earth Centred Earth Fixed (ECEF) Geodetic XYZ coordinates

• resolution_min (int, optional, default=0) – The minimum DGGS resolution to search for zones that are within the search_box.

• resolution_max (int, optional, default=1) – The maximum DGGS resolution to search for zones that are within the search_box.

• get_all_cells_flag (boolean, required) – A Flag to select whether or not to return all zones of the Surface DGGS.

• construct_cells_flag (boolean, required) – A Flag to select whether or not to construct zones of the Surface DGGS if they haven’t already been created and registered.

• equalise_cells_flag (boolean, required) – A Flag to select whether or not to equalise the surface area of the created Surface DGGS zone

• generate_cell_mesh_flag (boolean, optional, default=False) – A Flag to select whether or not to construct a 3D mesh of the DGGS zones once they have been found.

• construct_cells_key (str, optional, default="") – Unique identifier for the task.

• cell_df (JSON, optional, default={}) – JSON representation of a Pandas DataFrame containing the core metadata of a DGGS Zone

• cell_verts_df (JSON, optional, default={}) – JSON representation of a Pandas DataFrame containing the vertices metadata of a DGGS Zone

• cell_edges (JSON, optional, default={}) – JSON representation of a Pandas DataFrame containing the zone edge metadata of a DGGS Zone

• fixed_edges (array-like, optional, default=[]) – Array/List containing the fixed zone edges of a DGGS Zone

• colatitudes (array-like, optional, default=[]) – Array/List of colatitudes of each zone Edge. The colatitude measures the amoung of tilt/rotation of a Zone Edge about an axis through the two end vertices of that edge.

Returns:

output_dict – JSON compatible dictionary object is returned with the results of the sub-task(s).

Return type:

JSON

src.util.dggs_operations.initialise_zones_task(task_params)

Note

Function Details

Celery Task to search for child coverage zones for a polygon in parallel.

Parameters:

task_params (JSON, required) –

JSON representation of Task Parameters.

{'parent_cell_id' : "A123",
'searchBox': [[1231.0,-2132342.0,0.0],
              [-3334234.2342, 5534343.000, -123123.0],
              [2234234.00, 12312333.00334, 234233.05],
              [44345234, -12312333.0,550535.0],
              [1231.0,-2132342.0,0.0]]
'res' : 5,
'dggs_ID' : TerraNexus_Tesseract_e065a3f5b27a8795,
'equalise_cells_flag' : True}

Returns:

output_dict – JSON compatible dictionary object is returned with details of the tested DGGS Zone.

Return type:

JSON

src.util.dggs_operations.rotate_dae_model_task(dggs_specs={}, rotation_axis=[], rotation_angle=0, mesh=[], model_file='')

Note

Function Details

Celery Task to rotate a COLLADA “DAE” mesh model in parallel. This enables any “arbitrary” model object to be oriented within a “real-world” spatial context on the globe.

Parameters:

• dggs_specs (JSON, required) –

  JSON representation of DGGS instance specification. For example

  {'resolution_max':      15,
   'refinement_ratio':    9,
   'max_radius_multiple': 2,
   'shell_dggs_type':    'TerraNexus',
   'base_polyhedron':    'cube',
   'cell_type':          'square',
   'earth_model':        'GRS80',
   'a':                  6378137.0,
   'f':                  1.0,
   'ref_lon':            0.0,
   'ref_lat':            0.0}
  

• rotation_axis (array-like, required) – The rotation axis defined in vector notation.

• rotation_angle (float, required) – The angle to rotate around the “rotation_axis” in radians

• mesh (array-like, required) – The mesh array to rotate around the rotation axis.

• model_file (str, required) – The filename to store the “rottated” model to. This parameter is deprecated and will be replaced with a database drive solution.

Returns:

output_dict – JSON compatible dictionary object is returned with details of the tested DGGS Zone.

Return type:

JSON

src.util.dggs_operations.get_zoneTile_task(task_params)

Note

Function Details

Celery Task to get DGGS Zone Tiles (i.e. DGGS Zones containing multiple data objects) in parallel.

Parameters:

task_params (JSON, required) –

JSON representation of Task Parameters.

{'parent_cell_id' : "A123",
'searchBox': [[1231.0,-2132342.0,0.0],
              [-3334234.2342, 5534343.000, -123123.0],
              [2234234.00, 12312333.00334, 234233.05],
              [44345234, -12312333.0,550535.0],
              [1231.0,-2132342.0,0.0]]
'res' : 5,
'dggs_ID' : TerraNexus_Tesseract_e065a3f5b27a8795,
'equalise_cells_flag' : True}

Returns:

output_dict – JSON compatible dictionary object is returned with details of the tested DGGS Zone.

Return type:

JSON

src.util.dggs_operations.get_child_zone_coverage_task(task_params)

Note

Function Details

Celery Task to get DGGS Zone Coverages (i.e. DGGS Zones contained by a data feature) in parallel.

Parameters:

task_params (JSON, required) –

JSON representation of Task Parameters.

{'parent_cell_id' : "A123",
'searchBox': [[1231.0,-2132342.0,0.0],
              [-3334234.2342, 5534343.000, -123123.0],
              [2234234.00, 12312333.00334, 234233.05],
              [44345234, -12312333.0,550535.0],
              [1231.0,-2132342.0,0.0]]
'res' : 5,
'dggs_ID' : TerraNexus_Tesseract_e065a3f5b27a8795,
'equalise_cells_flag' : True}

Returns:

output_dict – JSON compatible dictionary object is returned with details of the tested DGGS Zone.

Return type:

JSON

src.util.dggs_operations.get_child_zone_coordinates_task(task_params)

Note

Function Details

Celery Task to get child DGGS Zone Coordinates (i.e. DGGS Zones that contain a single point or node of a feature at the next DGGS resolution level) in parallel.

Parameters:

task_params (JSON, required) –

JSON representation of Task Parameters.

{'parent_cell_id' : "A123",
'searchBox': [[1231.0,-2132342.0,0.0],
              [-3334234.2342, 5534343.000, -123123.0],
              [2234234.00, 12312333.00334, 234233.05],
              [44345234, -12312333.0,550535.0],
              [1231.0,-2132342.0,0.0]]
'res' : 5,
'dggs_ID' : TerraNexus_Tesseract_e065a3f5b27a8795,
'equalise_cells_flag' : True}

Returns:

output_dict – JSON compatible dictionary object is returned with details of the tested DGGS Zone.

Return type:

JSON

src.util.dggs_operations.map_featureData_task(task_params)

Celery Task to map Feature Objects to a DGGS.

Parameters:

task_params (JSON, required) – The task parameters for this task

Returns:

output_dict – JSON compatible dictionary with task outputs

Return type:

JSON

src.util.dggs_operations.reorganiseData_task(task_params)

Celery Task to reorganise the spatial data of the input dataset for consistency with TerraNexus functions.

Parameters:

task_params (JSON, required) – The task parameters for this task

Returns:

output_dict – JSON compatible dictionary with task outputs

Return type:

JSON

src.util.dggs_operations.processData_task(job_id, datafilename, data_type='Feature', provChain={}, callback_json={})

Data processing function to process an input dataset and map it to a DGGS.

Parameters:

• job_id (str, required) – Unique Identification of the Job being Processed

• datafilename (str, required) – Name of the raw Data File to be processed

• data_type (str, optional, default = "Feature") – Broad Data Type of the data being processed (e.g. Feature, Coverage, Image, etc…)

• provChain (Dictionary-Like, optional, default = {}) – Provenance Chain information necessary to log Provenance.

• callback_json (JSON, optional, default = {}) – Callback message (in JSON format) to send to the Process Callback Listeners.

util.dggs_util module

Note

Module Details

DGGS General Utility Script

This Module contains the Class Object that defines the general DGGS functions to support the operation of both Surface and Volumetric DGGS Instances registered to the TerraNexus Web Application Infrastructure.

This module, coupled with the surface_dggs.scripts.surface_dggs_setup module, the volumetric_dggs.scripts.volumetric_dggs_setup module as well as the common utility functions defined in the util.dggs_operations module, represents a production grade implementation of Patent WO2018/136996 Purss and Pangaea-Innovations-Pty.-Ltd. [2017]

Created: 21 April 2017

@author: Dr. Matthew B.J. Purss

class src.util.dggs_util.DGGS_Tools(a0=6378137.0, f=0.003352810681182319, logfile='solid_earth_dggs.log')

Bases: object

Note

Class Details

DGGS Tools Class Structure

This Class handles the general utility functions that operate across both Surface and Volumetric DGGS instances registered on the TerraNexus Web Application Infrastructure.

Parameters:

• a0 (float, optional, default=6378137.0) – The semi-major ellipsoidal axis radius of the earth model in metres. This is equivalent to the equatorial radius

• f (float, optional, default=1.0) – The earth flattening ratio that defines the degree of flattening of the Earth at its poles. This determines the semi-minor ellipsoidal axis radius of the earth model (i.e. the polar radius).

• logfile (str, optional, default='solid_earth_dggs.log') – The “filename” for the logfile - this is now a MongoDB database table/collection name that is created inside the “logs” MongoDB Database.

_gpu_dist_to_polygon_edge(poly, pnts_df)

Note

Function Details

Utility Function to compute the distance from a set of point locations to the nearest polygon edge.

Parameters:

• poly (array-like, required) – The polygon defined as an array of vertices in ECEF XYZ Coordinates.

• pnts_df (Pandas DataFrame, required) – The Pandas DataFrame containing the Point Location information.

Returns:

pnts_df – updated pnts_df Pandas DataFrame with the computed distance from each point to the nearest edge of the polygon.

Return type:

Pandas DataFrame

apply_morton_search(search_box, search_res_min, search_res_max, mongodbname='', surface_DGGS_flag=True)

Note

Function Details

Utility Function to apply a morton code search for DGGS Zones.

This is a critical component of the TerraNexus Volumetric DGGS infrastructure and follows the method described in the Patent Purss and Pangaea-Innovations-Pty.-Ltd. [2017]

Parameters:

• search_box (array-like, required) – The Search Box in Earth Centred Earth Fixed (ECEF) Geodetic XYZ coordinates

• search_res_min (int, required) – The minimum level of the DGGS resolution hierarchy tree to search

• search_res_max (int, required) – The minimum level of the DGGS resolution hierarchy tree to search

• mongodbname (str, required) – The name of the “MongoDB” database contianing the DGGS specification. This is also the dggrsID

• surface_DGGS_flag (boolean, optional, default=True) – Flag to select whether or not the Morton Code Search is to be conducted on a Surface DGGS or Volumetric DGGS instance.

Returns:

registered_zones – Array/List of Registered Zones returned from the Morton-code index search.

Return type:

array-like

check_if_collections_exists(db, tablenames)

Note

Function Details

Utility function to check if a given list of MongoDB Database Collections (i.e. tables) exists.

Parameters:

• db (str, required) – The name of the MongoDB Database to test for.

• tablenames (array-like, required) – Array/List of MongoDB Database Collection names to test for existence

Returns:

check_tables_result – A Flag that indicates whether, or not, the given database collections exists.

Return type:

boolean

check_if_mongodb_exists(db)

Note

Function Details

Utility function to check if a given MongoDB Database exists.

Parameters:

db (str, required) – The name of the MongoDB Database to test for.

Returns:

db_status – A Flag that indicates whether, or not, the given database exists.

Return type:

boolean

circumcircle(points, simplex)

Note

Function Details

Utility function to compute the circumcircle (a circle touching all the vertices of a triangle or polygon.) of an individual spatial object (simplex)

Note

This function is used to compute complex (concave) hull meshes around a set of points. The Circumcircle is used to test whether or not a mesh facet or voxel can be refined to better represent the shape of the data.

Parameters:

• points (array-like, required) – Array/List of point locations to test for the circumcircle of a mesh object (simplex)

• simplex (array-like, required) – Array/List of a mesh object facet - i.e. a triangular facet.

Returns:

circumcirle – A tuple object containing the Centre and Radius of the Circumcircle as items.

Return type:

tuple

decode_morton_id(mortonCode, normalise_height=False)

Note

Function Details

Utility function to decode a point position from a TerraNexus Ellipsoidal Morton Code Identifier following the method set out in Patent WO2018/136996 Purss and Pangaea-Innovations-Pty.-Ltd. [2017]

Parameters:

• mortonCode (string, required) – string object representing a TerraNexus Ellipsoid Morton Code Identifier.

• normalise_height (boolean, optional, default=False) – Flag to select whether, or not, to return the height value relative to the Earth Surface or Core.

Returns:

point_position – Array/List of point location(s) in Geodetic Longitude, Latitude, Height coordinates.

Return type:

array-like

delete_data_from_mongo(db, tablename, query={})

Note

Function Details

Utility function to delete data stored in a MongoDB Database Collection

Parameters:

• db (str, required) – The name of the MongoDB Database to return data from

• tablename (str, required) – MongoDB Database Collection name to return data from

• query (JSON, optional, default={}) – The query to send to the MongoDB Database Collection.

Returns:

deleted_data_count – The number of data records deleted.

Return type:

int

drop_mongo_table(db='test_db', tablename='test_table')

Note

Function Details

Utility function to drop (delete) a MongoDB Database or collection

Parameters:

• db (str, required) – The name of the MongoDB Database to drop

• tablename (str, required) – MongoDB Database Collection name to drop

ecef2ll(pnt, a=0.0, f=-1.0, high_precision=False)

Note

Function Details

Function to perform the conversion from Cartesian X,Y,Z coordinates to geodetic Lon,Lat,Height following the equations published in

IOGP Publication 373-7-2 (available from www.epsg.org) IOGP [2021]

Parameters:

• pnt (array-like, required) – The point, or points, to be converted from ECEF XYZ Coordinates to Geodetic Longitude, Latitude, Height Coordinates.

• a (float, optional, default=6378137.0) –

  The semi-major ellipsoidal radius of the Earth Model. For ellipsoids of rotation (like the Earth Model) this is equivalent to the Equatorial Radius from the centre of the Earth.

  ..note

  if "a" is omitted or set to 0.0 then the Class instance of "a" (i.e. `self.a`) will be used
  

• f (float, optional, default=-1.0) –

  The Earth Flattening Ratio. The Ratio of flattening of the polar axes verses the equatorial axes of the Earth Model to properly define the ellipsoid of rotation.

  ..note

  if "f" is omitted or set to -1.0 then the Class instance of "f" (i.e. `self.f`) will be used
  

• high_precision (boolean, optional default=False) – Flag to select whether or not to perform the conversion using the mpmath library to enable arbitrary precision computations that go beyond the standard python floating point precision.

Returns:

convertedData – A numpy array containing the converted point, or points.

Return type:

array-like

encode_morton_id(lon, lat, height)

Note

Function Details

Utility Function to encode a point position into a TerraNexus Ellipsoidal Morton Code Identifier following the method set out in Patent WO2018/136996 Purss and Pangaea-Innovations-Pty.-Ltd. [2017]

Parameters:

• lon (float) – longitude coordinate value in decimal degrees.

• lat (float) – latitude coordinate value in decimal degrees.

• height (float) – ellipsoidal height in metres

Returns:

mortonCode – string object representing the TerraNexus Ellipsoid Morton Code Identifier for the given position.

Return type:

string

ensure_index_exists(collection, key: str | List[Tuple[str, int]], sort_direction: str = 'ASCENDING', **kwargs)

Check if an index exists for the given key(s) and create it if it doesn’t exist.

Args:

collection: PyMongo collection object key: Either a string for single field index or list of (field, direction) tuples for compound **kwargs: Additional index options (unique, sparse, etc.)

Returns:

str: Name of the index (existing or newly created)

get_alpha_surface(mesh, alpha_radius=1.0)

Note

Function Details

Utility function to compute the Alpha Mesh Surface of a given Mesh for a given alpha-radius

Note

the value of the alpha radius determines the effort and complexity of the mesh that produces an alpha shape that encloses all points

Parameters:

• mesh (array-like, required) – The Mesh object to create an alpha-mesh from.

• alpha_radius (float, optional, default=1.0) – The radius threshold of all circumcircles to determine whether or not to refine part of the mesh.

Returns:

alpha_mesh – The Alpha Mesh computed by refining the specified Mesh using a given alpha radius.

Return type:

array-like

get_boundary_points(theta_v1, theta_v2, m, r, s, ellipse_a, ellipse_b, v1, v2, a, f, num_subdivisions=20, webgl_flag=False)

Note

Function Details

Function to compute the points along the DGGS Zone Boundary Path.

Parameters:

• theta_v1 (float, required) – the angle of rotation from the centre of the ellipse of intersection (m) to vertex 1 (V1).

• theta_v2 (float, required) – the angle of rotation from vertex 1 (V1) to vertex 2 (V2) relative to the centre of the ellipse of intersection (m).

• m (array-like, required) – point location of the centre of the ellipse of intersection “m” in ECEF XYZ Coordinates

• r (array-like, required) – unit vector from the centre of the ellipse of intersection “m” along the semi-major axis of the ellipse in ECEF XYZ Coordinates

• s (array-like, required) – unit vector from the centre of the ellipse of intersection “m” along the semi-minor axis of the ellipse in ECEF XYZ Coordinates

• ellipse_a (float, required) – semi-major axis radius of the ellipse of intersection from its centre (m) to the ellipsoid surface.

• ellipse_b (float, required) – semi-minor axis radius of the ellipse of intersection from its centre (m) to the ellipsoid surface.

• v1 (array-like, required) – list or numpy array with the ECEF (XYZ) coordinates for vertex “1”.

• v2 (array-like, required) – list or numpy array with the ECEF (XYZ) coordinates for vertex “2”.

• a (float, required) – semi-major axis radius of the ellipsoid/spheroid

• f (float, required) – flattening factor of the ellipsoid/spheroid. For a Sphere f = 0.0

• num_subdivisions (integer, optional) – Number of points to define along the arc from V1 to V2. [Default = 20].

• webgl_flag (boolean, optional) – flag to scale the returned coordinate values to a valid range to display using webgl. [Default = False]

Returns:

pnts_list – An array/list of evenly spaced points along the ellipsoidal arc along a DGGS Zone Boundary Edge.

Return type:

array-like

get_cell_area(cell_df=[], cell_verts_df=[], cell_edges=[], fixed_edges=[], colatitudes=[], a=0.0, f=-1.0, isolatitude_flag=[], dggs_type='TerraNexus')

Note

Function Details

Utility Function to compute the surface area of a DGGS Zone on an ellipsoid following the method described in Sjöberg [2006].

Parameters:

• cell_df (Pandas DataFrame, required) – A Pandas DataFrame containing the core DGGS Zone metadata

• cell_verts_df (Pandas DataFrame, required) – A Pandas DataFrame containing the DGGS Zone vertices

• cell_edges (Pandas DataFrame, required) – A Pandas DataFrame containing the DGGS Zone edge specification

• fixed_edges (array-like, optional, default=[]) – A list of fixed edges - referenced by the edge index.

• colatitudes (array-like, optional, default=[]) – A list of edge colatitudes - referenced by the edge index.

• a (float, optional, default=6378137.0) –

  The semi-major ellipsoidal radius of the Earth Model. For ellipsoids of rotation (like the Earth Model) this is equivalent to the Equatorial Radius from the centre of the Earth.

  ..note

  if "a" is omitted or set to 0.0 then the Class instance of "a" (i.e. `self.a`) will be used
  

• f (float, optional, default=-1.0) –

  The Earth Flattening Ratio. The Ratio of flattening of the polar axes verses the equatorial axes of the Earth Model to properly define the ellipsoid of rotation.

  ..note

  if "f" is omitted or set to -1.0 then the Class instance of "f" (i.e. `self.f`) will be used
  

• isolatitude_flag (list, optional, default=[]) – List of edges that are isolatitudinal (i.e. run parallel to the lines of latitude). This flag is used to determine if the simple or comprehensive computation of the ellipsoidal arc between the two vertices along the DGGS Zone Edge should be used.

• dggs_type (str, optional, default='TerraNexus') – The Type of DGGS Instance.

Returns:

cell_area_package – A Dictionary/JSON object containing the following details:

{'cell_area':cell_area,
'cell_edges': cell_edges,
'ext_angles': ext_angles,
'gaussian_curvature': gaussian_curvature,
'new_edge_colatitudes': new_edge_colatitudes
}

Return type:

JSON

get_cell_id(lon, lat, height, resolution)

Note

Function Details

Utility function to combine an ellipsoidal morton-code (encoded from a geodetic coordinate location) with a binary representation of the DGGS resolution to create a TerraNexus Morton-Code based DGGS Zone Identifier. This follows the procedure set out in Patent WO2018/136996 Purss and Pangaea-Innovations-Pty.-Ltd. [2017]

Parameters:

• lon (float, required) – longitude coordinate value in decimal degrees.

• lat (float, required) – latitude coordinate value in decimal degrees.

• height (float, required) – ellipsoidal height in metres

• resolution (int, required) – The DGGS resolution to construct the ellipsoidal shell tree on.

Returns:

composite_id – A composite Morton-Code + resolution identifier which represents a Morton-Code based DGGS ZoneId.

Return type:

str

get_centroid(verts, earth_model='GRS80', a=0.0, f=-1.0)

Note

Function Details

Utility Function to compute the centroid from a set of point locations. These may be either isolated point locations or a polygon structure.

Parameters:

• verts (array-like, required) – A list/array of point locations in ECEF XYZ coordinates.

• earth_model (ste, optional, default='GRS80') – The Earth Model to project the centroid location onto.

• a (float, optional, default=6378137.0) –

  The semi-major ellipsoidal radius of the Earth Model. For ellipsoids of rotation (like the Earth Model) this is equivalent to the Equatorial Radius from the centre of the Earth.

  ..note

  if "a" is omitted or set to 0.0 then the Class instance of "a" (i.e. `self.a`) will be used
  

• f (float, optional, default=-1.0) –

  The Earth Flattening Ratio. The Ratio of flattening of the polar axes verses the equatorial axes of the Earth Model to properly define the ellipsoid of rotation.

  ..note

  if "f" is omitted or set to -1.0 then the Class instance of "f" (i.e. `self.f`) will be used
  

Returns:

centroid_package – The centroid information as a tuple containing the following items:

• centroid_ll (the centroid location in geodetic longitude, latitude, height coordinates);

• centroid_ecef (the centroid location in ECEF XYZ coordinates);

• unit_centroid (the centroid represented as a unit vector);

• centroid_kt (the magnitude of the vector from the centre of the Earth to it’s surface along the centroid unit-vector direction).

Return type:

tuple

get_child_cells(cell_id='', refine_cell_params={}, number_of_levels=1)

Note

Function Details

Function to get the child Surface DGGS Zones (Cells) from a specified parent Zone.

Parameters:

• dggrsID (str, required) – The DGGS Identifier.

• cell_id (str, required) – The topological Surface DGGS ZoneID.

• number_of_levels (int, optional, default=1) – The number of levels down from the selected DGGS Zone to find Parent Zones.

Returns:

parent_cell_ids – The list of Parent DGGS Zones found.

Return type:

list-list

get_child_voxels(voxel_id='', refine_cell_params={}, number_of_levels=1)

Note

Function Details

Function to get the child Volumetric DGGS Zones (Voxels) from a specified parent Zone.

Parameters:

cell_id (str, required) – The DGGS ZoneId of the Parent DGGS Zone

refine_cell_params: JSON, required

parameter dictionary/JSON object that drives this function. The structure of this JSON object is as follows:

refine_cell_params = {'id':                    <zoneId>,
                      'refinement_ratio':      <refinement_ratio>,
                      'parent_cell_centroid':  [x, y, z],
                      'parent_cell_verts':    [[x1, y1, z1],
                                               [x2, y2, z2],
                                               ...,
                                               [xn, yn, zn]],
                      'a0':                    <semi-major radius>,
                      'cell_type':             <cell_type>,
                      'base_polyhedron':       <base_polyhedron>}

number_of_levels: int, optional, default=1

The number of DGGS Resolution Levels to go down to find child Zones.

Returns:

child_cell_dict – The Dictionary/JSON object containing the structural details of the returned child DGGS Zones.

Return type:

JSON

get_config()

Note

Function Details

Internal function to get or create the TerraNexus OGC API Server Configuration settings.

get_ellipsoid_shells(mongodbname='', dggs_params={}, res=0, shell_min=0, shell_max=-1, a_min=6378136.0, a_max=6378137.0, dggs_voxel=False, chunksize=100000)

Note

Function Details

Utility Function to compute the ellipsoid shells for a given set of ellipsoid specifications.

This is a critical component of the TerraNexus Volumetric DGGS infrastructure and follows the method of construction set out in the Patent Purss and Pangaea-Innovations-Pty.-Ltd. [2017]

Parameters:

• mongodbname (str, required) – The name of the “MongoDB” database contianing the DGGS specification. This is also the dggrsID

• dggs_params (JSON, required) –

  JSON/Dictionary object with the specifications for the selected Surface DGGS Instance. For example

  {'resolution_max':      15,
   'refinement_ratio':    9,
   'max_radius_multiple': 2,
   'shell_dggs_type':    'TerraNexus',
   'base_polyhedron':    'cube',
   'cell_type':          'square',
   'earth_model':        'GRS80',
   'a':                  6378137.0,
   'f':                  1.0,
   'ref_lon':            0.0,
   'ref_lat':            0.0}
  

• res (int, optional, default=0) – The DGGS resolution to construct the ellipsoidal shell tree on.

• shell_min (int, optional, default=0) –

  The index of the minimum (lowest) ellipsoid shell to construct and store.

  Note

  The “0” shell index is the surface of the Earth Model

• shell_max (int, optional, default=-1) –

  The index of the maximum (highest) ellipsoid shell to construct and store.

  Note

  Negative shell indexes extend away from the surface of the Earth Model

• a_min (float, optional, default=1.0) – The minimum semi-major ellipsoidal axis radius of the earth model in metres to define the ellipsoidal shells of the Volumetric DGGS instance

• a_max (float, optional, default=1.0) – The maximum semi-major ellipsoidal axis radius of the earth model in metres to define the ellipsoidal shells of the Volumetric DGGS instance

• dggs_voxel (boolean, optional, default=False) – A Flag to select whether or not to use voxel derived “shell_indices” to determine the upper and lower ellipsoid shell bounds or the a_min and a_max ellipsoidal radii

• chunk_size (int, optional, default=100000) – Chunk size to use to throttle the process to ensure it does not exceed the hardware memory resource capacity.

Returns:

ellipsoidal_shells – The ellipsoidal shells as a Pandas DataFrame.

Return type:

Pandas DataFrame

get_existing_index_name(collection, target_key: List[Tuple[str, int]]) → str

Get the name of an existing index that matches the target key.

Args:

collection: PyMongo collection object target_key: List of (field, direction) tuples representing the index key

Returns:

str: Name of the existing index, or None if not found

get_neighbouring_cells(dggrsID, cell_id='', get_edge_neighbours=True, get_vertex_neighbours=True)

Note

Function Details

Function to get the neighbouring Surface DGGS Zones around a specified Zone.

Parameters:

• cell_id (str, required) – The topological Surface DGGS ZoneID.

• get_edge_neighbours (boolean, optional, default=True) – A Flag to select whether or not to return `Edge` Neigbhours

• get_vertex_neighbours (boolean, optional, default=True) – A Flag to select whether or not to return `Vertex` Neigbhours

Returns:

neigbouring_cell_ids – The list of Neighbouring DGGS Zones found.

Return type:

list-like

get_neighbouring_voxels(cell_id='', number_of_rings=1, get_edge_neighbours=True, get_vertex_neighbours=True)

[NOT IMPLEMENTED YET]

get_parent_voxels(refinement_ratio, voxel_id, number_of_levels=1, return_voxel_specs=False)

Note

Function Details

Function to get the parent Volumetric DGGS Zones (Voxels) from a specified child Zone.

Parameters:

• refinement_ratio (int, required) – The Refinement Ratio governing the number of Child Zones to each Parent Zone.

• voxel_id (str, required) – The topological Volumetric DGGS ZoneID. This has the structure <cellId>_<shell_id>

• number_of_levels (int, optional, default=1) – The number of levels up from the selected DGGS Zone to find Parent Zones.

• return_voxel_specs (boolean, optional, default=False) – Flag to return the Volumetric DGGS Zone Specification for the Parent DGGS Zones retrieved.

Returns:

parent_voxels – The list of Parent DGGS Zones found.

Return type:

list-list

get_prov_from_mongo(entityID, explicit_search=False)

Retrieve provenance information from MongoDB for a given entity.

Args:

entityID (str): The ID of the entity to retrieve provenance for explicit_search (bool): If True, search for exact match, else use regex

Returns:

dict: A dictionary containing provenance information

get_registered_cells(mongodbname, search_type='All_cells', search_box_label='', search_res_min=0, search_res_max=-1, cell_ids=[], search_box={}, search_radius=0, ref_location=[], m2deg_factor=111325, chunk_size=10000, db_bypass=False)

Note

Function Details

Utility Function to search and retrieve registered Surface DGGS Zones from the MongoDB Database.

Parameters:

• mongodbname (str) – database name for the DGGS

• search_type (str) –

  type of search to perform valid options are:

  ['All_cells','cellID','res','search_box']
  * All = get all registered cells
  * cellID = get all registered cells for the given list of cellIDs
  * morton_code = get all registered zones for the given list of morton_codes
  * res = get all registered cells for the given range of resolutions
  * search_box = use a search box to perform a Morton_code range search
  * search_polygon = use a polygon to perform a Morton_code range search
  * range_search = use a distance search from a reference location
  

• search_box_label (str, optional) – label of the search box to use for bounding box and polygon searches

• search_res_min (int, optional) – minimum resolution to search. [Default = 0]

• search_res_max (int, optional) – maximum resolution to search. [Default = 1]

• cell_ids (list, optional) – list of cellIDs to search for via a zone index lookup [Default = []]

• search_box (list-like, optional) – a list/array containing the coordinates for the search box, or search polygon, vertices [Default = []]

• search_radius (float, optional) – search radius (in metres) from a reference location to perform a radial search on [Default = 0]

• ref_location (array-like, optional) – list or numpy array with the [lon, lat, height] of the reference location for range search [Default = []]

• m2deg_factor (float, optional) – Approximate metres-to-degrees conversion factor. [Default = 111325]

• chunk_size (int, optional) – chunk size to limit the number of records returned. [Default = 10000]

• db_bypass (boolean, optional) – flag to bypass full database query result extraction. [Default = False]

Returns:

registered_zones – Array/List of Registered Zones returned from the search.

Return type:

array-like

get_registered_voxels(mongodbname, search_type='All_voxels', search_res_min=0, search_res_max=0, voxel_ids=[], cell_ids=[], search_box={}, search_radius=0, height_min=-1, height_max=1, ref_location=[], m2deg_factor=111325)

Note

Function Details

Utility Function to search and retrieve registered Volumetric DGGS Zones from the MongoDB Database.

Parameters:

• mongodbname (str) – database name for the DGGS

• search_type (str) –

  type of search to perform valid options are

  * All = get all registered cells
  * cellID = get all registed cells for the given list of cellIDs
  * res = get all registered cells for the given range of resolutions
  * search_box = use a search box to perform a Morton_code range search
  * range_search = use a distance search from a reference location
  

• search_box_label (str, optional) – label of the search box to use for bounding box and polygon searches

• search_res_min (int, optional) – minimum resolution to search. [Default = 0]

• search_res_max (int, optional) – maximum resolution to search. [Default = 1]

• cell_ids (list, optional) – list of cellIDs to search for via a zone index lookup [Default = []]

• search_box (dictionary, optional) – a dictionary containing the coordinates for the search box vertices [Default = {}]

• search_radius (float, optional) – search radius (in metres) from a reference location to perform a radial search on [Default = 0]

• ref_location (array-like, optional) – list or numpy array with the [lon, lat, height] of the reference location for range search [Default = []]

• m2deg_factor (float, optional) – Approximate metres-to-degrees conversion factor. [Default = 111325]

Returns:

registered_zones – Array/List of Registered Zones returned from the search.

Return type:

array-like

get_search_radius(search_radius, ref_location_xyz, num_edges=360)

Note

Function Details

Function to compute and return a circular search polygon specified by a given search radius about a reference location.

Parameters:

• search_radius (float, required) – The Search Radius in metres.

• ref_location_xyz (list, required) – The Reference Location in ECEF XYZ Coordinates

• num_edges (integer, optional, default=360) – The number of edges for the output search polygon. [Minimum = 3]

Returns:

search_box – The Computed Search Polygon inscribed by the Search Radius.

Return type:

list-like

get_small_ellipse(v1, v2, colatitude=-1, a=0.0, f=-1.0, general_ellipse=False, isolatitude=False, cell_id='<ID>', test_flag=False)

Note

Function Details

Function to compute the small circle on the surface of the ellipsoid that passes through the vertices of the platonic solid.

follows the derivation of the intersection of a plane and Ellipsoid by Klein [2012], modified to properly describe the no of a great circle (normal) intercept with the ellipsoid surface.

Parameters:

• v1 (array-like) – list or numpy array with the ECEF (XYZ) coordinates for vertex “1”.

• v2 (array-like) – list or numpy array with the ECEF (XYZ) coordinates for vertex “2”.

• colatitude (float, optional) – the colatitude (angle of deflection from pi() radians) of the ellipsoidal arc drawn from v1 to v2. [Default = -1]

• a (float) – semi-major axis radius of the ellipsoid/spheroid

• f (float) – flattening factor of the ellipsoid/spheroid. For a Sphere f = 0.0

• general_ellipse (boolean, optional) – flag to compute a general ellipse along the semi-major axis (used for testing and validation). [Default = False]

• isolatitude (boolean, optional) – flag to compute a general ellipse along the semi-minor axis.

• cell_id (str, optional) – label for the dggs zone this curve is associated to (if provided)

• test_flag (boolean, optional) – flag to compute validation tests. [Default = False]

Returns:

small_ellipse – A Dictionary/JSON object that contains the specification of the requested small ellipse

Return type:

JSON

get_unit_vectors(verts, single_pnt=True)

Note

Function Details

Utility Function to return the unit vectors of any vector.

Parameters:

• verts (array-like, required) – A list/array of vectors to convert to unit vectors. In the context of TerraNexus every coordinate location in 3D space is a vector from the origin to that location.

• single_pnt (boolean, optional, default=True) – A Flag to select whether or not to convert a single vector or multiple vectors in a single operation.

Returns:

unit_vector – The computed unit vectors of the input vectors.

Return type:

array-like

grouper(iterable, n, fillvalue=None)

Note

Function Details

Internal function to collect data into fixed-length chunks or blocks

handle_uploaded_file(upload_dir, f)

Note

Function Details

Utility function to handle files uploaded to the TerraNexus Server and store them in a common location that can be accessed by all relevant TerraNexus pods.

Parameters:

• upload_dir (str, required) – The path/directory to store the new uploaded file in.

• f (file object, required) – The file object containing the uploaded file.

index_exists(collection, target_key: List[Tuple[str, int]]) → bool

Check if an index exists for the given key specification.

Args:

collection: PyMongo collection object target_key: List of (field, direction) tuples representing the index key

Returns:

bool: True if index exists, False otherwise

init_mongo_collection(db='test_db', tablename='test_tbl', shard_db=False, capped=False, capped_size=10737418240, index_spec=[])

Note

Function Details

Utility function to Initialise a MongoDB Database collection

Parameters:

• db (str, required) – The name of the MongoDB Database to initialise

• tablename (str, required) – MongoDB Database Collection name to initialise

• shard_db (boolean, optional, default=True) – Flag to select whether, or not, to create a sharded database across the MongoDB Cluster Infrastructure.

• capped (boolean, optional, default=True) – Flag to select whether, or not, to cap the size of this MongoDB Database Collection.

• capped_size (int, optional, default=10737418240) – Maximum disk size limit for this MongoDB database collection in bytes.

• index_spec (array-like, optional, default=[]) – Array/List that contains the specification of which database fields will be indexed and in what direction (ASCENDING or DESCENDING)

init_mongo_database(db='test_db', shard_db=False)

Note

Function Details

Utility function to Initialise a MongoDB Database

Parameters:

• db (str, required) – The name of the MongoDB Database to return data from

• shard_db (boolean, optional, default=False) – Flag to select whether, or not, to create a sharded database across the MongoDB Cluster Infrastructure.

list_all_indexes(collection)

List all indexes in the collection with their details.

Args:

collection: PyMongo collection object

ll2ecef(pnt, a=0.0, f=-1.0, high_precision=False)

Note

Function Details

Function to perform the conversion from geodetic Lon,Lat,Height to Cartesian X,Y,Z coordinates following the equations published in

IOGP Publication 373-7-2 (available from www.epsg.org) IOGP [2021]

Parameters:

• pnt (list-like, required) – The point, or points, to be converted from Geodetic Longitude, Latitude, Height Coordinates to ECEF XYZ Coordinates

• a (float, optional, default=6378137.0) –

  The semi-major ellipsoidal radius of the Earth Model. For ellipsoids of rotation (like the Earth Model) this is equivalent to the Equatorial Radius from the centre of the Earth.

  ..note

  if "a" is omitted or set to 0.0 then the Class instance of "a" (i.e. `self.a`) will be used
  

• f (float, optional, default=-1.0) –

  The Earth Flattening Ratio. The Ratio of flattening of the polar axes verses the equatorial axes of the Earth Model to properly define the ellipsoid of rotation.

  ..note

  if "f" is omitted or set to -1.0 then the Class instance of "f" (i.e. `self.f`) will be used
  

• high_precision (boolean, optional default=False) – Flag to select whether or not to perform the conversion using the mpmath library to enable arbitrary precision computations that go beyond the standard python floating point precision.

Returns:

convertedData – A numpy array containing the converted point, or points.

Return type:

array-like

mesh_voxel(dggs_voxel, dggs_type='TerraNexus')

[NOT YET IMPLEMENTED]

nested_dictionary_search(search_dict, field, path=[])

Note

Function Details

Internal function to perform a query across a “Nested” Dictionary or JSON object.

Parameters:

• search_dict (JSON, required) – The Dictionary or JSON object to search

• field (str, required) – The key to search the Dictionary for

• path (list-like, optional, default=[]) – The path within the Dictionary or JSON tree structure to iteratively search for the “field”.

Returns:

search_results – Tuple containing the value (if it exists) and the path to the “field” the Dictionary was searched for.

Return type:

tuple

point_inside_cell(pnt, dggs_cell, rec_id=[], df_index=[], dggs_type='TerraNexus', planar_search_mode_flag=True)

Note

Function Details

Utility Function to test if a point location is inside a DGGS Zone.

Parameters:

• pnt (array-like, required) – The point, or points, to be tested in ECEF XYZ Coordinates.

• dggs_cell (Class-Object, required) – The DGGS_Cell class object instance for the DGGS Zone of interest.

• rec_id (array-like, optional, default=[]) – Array/List of record identifiers.

• df_index (array-like, optional, default=[]) – Array/List of Pandas DataFrame index values.

• dggs_type (str, optional, default='TerraNexus') – The Type of DGGS Instance.

• planar_search_mode_flag (boolean, optional, default=True) – Flag to perform the search based on the surface expression of the DGGS Zone or the planar expression (i.e. the plane formed by the vertices of the DGGS Zone).

Returns:

nearby_pnts_df – Pandas DataFrame with all points that are inside the specified DGGS Zone.

Return type:

Pandas DataFrame

point_inside_polygon(pnt, poly, df_flag=False, rec_id=[], method='ray_cast', gpu_flag=False, dateline_flag=False, sth_pole_flag=False, a=0.0, f=-1.0)

Note

Function Details

Utility Function to test if a set of point locations are inside a polygon.

Parameters:

• pnt (array-like, required) – The point, or points, to be tested in ECEF XYZ Coordinates.

• poly (array-like, required) – The polygon defined as an array of vertices in ECEF XYZ Coordinates.

• df_flag (boolean, optional, default=False) – Flag to select whether or not the input polygons are provided as Pandas DataFrames or Numpy Arrays.

• rec_id (array-like, optional, default=[]) – Array/List of record identifiers.

• method (str, optional, default='ray_cast') – Method to use to test if the point(s) are inside the polygon.

• gpu_flag (boolean, optional, default=False) – Flag to select whether or not to perform the calculation using GPU’s

• dateline_flag (boolean, optional, default=False) – Flag to select whether or not to test if the polygon crosses the 180 deg. Longitude line.

• sth_pole_flag (boolean, optional, default=False) – Flag to select whether or not to test if the South Pole is inside the polygon.

Returns:

nearby_pnts_df – Pandas DataFrame with all points that are inside the specified polygon.

Return type:

Pandas DataFrame

point_inside_solid(pnt, solid, solid_type='search_box', dggs_type='TerraNexus', idx=[], rec_id=[])

Note

Function Details

Utility function to test if a point is inside (or outside) a solid object represented by a mesh.

Parameters:

• pnt (array-like,) – array object containing 1 or more points to test.

• solid (Mesh object,) – The Solid model object to test if the point is inside. This is usually a triangular mesh in a particular mesh format.

• solid_type (string, optional) – type of solid object to compare points to:

• dggs_type (str, optional) – DGGS Type/Class to use for this operation [Default = TerraNexus]

• idx (array-like, optional) – Custom index for the point dataset.

Returns:

trimmed_pnt_df – Pandas DataFrame with the points that are inside the specified Model Object.

Return type:

Pandas DataFrame

point_inside_voxel(pnt, dggs_voxel, rec_id=[], df_index=[], shell_dggs_type='TerraNexus', planar_search_mode_flag=True)

Note

Function Details

Function to test if a point is inside a Volumetric DGGS Zone (Voxel)

Parameters:

• pnt (array-like, required) – The point, or points, to be tested in ECEF XYZ Coordinates.

• dggs_voxel (Class-Object, required) – The DGGS_Voxel class object instance for the DGGS Zone of interest.

• rec_id (array-like, optional, default=[]) – Array/List of record identifiers.

• df_index (array-like, optional, default=[]) – Array/List of Pandas DataFrame index values.

• dggs_type (str, optional, default='TerraNexus') – The Type of DGGS Instance.

Returns:

nearby_pnts_df – Pandas DataFrame with all points that are inside the specified Volumetric DGGS Zone.

Return type:

Pandas DataFrame

polygon_intersect(poly_1, poly_2, df_flag=False, poly_line_flag=False, poly_id='--', search_radius=-1)

Note

Function Details

Function to compute the intersection between two ploygons

Note

This tests whether polygon_1 and polygon_2 intersect using the Hessian Normal Form to describe a plane.

Reference: Weisstein

Parameters:

• poly_1 (array-like, required) – array/list of points defining the vertices of polygon-1 in ECEF XYZ Coordinates

• poly_2 (array-like, required) – array/list of points defining the vertices of polygon-2 in ECEF XYZ Coordinates

• df_flag (boolean, optional, default=False) – Flag to select whether or not the input polygons are provided as Pandas DataFrames or Numpy Arrays.

• poly_line_flag (boolean, optional, default=False) – Flag to select whether or not the 2nd polygon is a lineString rather than a Polygon.

• poly_id (str, optional, default='---') – Identifier for the process output.

• search_radius (float, optional, default=-1) –

  The Search Radius to apply to the search for planar intersects between polygon-1 and polygon-2

  ..note:

  if search_radius = -1 then compute a search radius from the size of the smallest polygon.
  

Returns:

polygon_intersect – A Flag indicating whether or not polygon-1 intersects with polygon-2

Return type:

boolean

polygon_intersects_cell(poly, dggs_cell, dggs_type='TerraNexus')

Note

Function Details

Function to test if a polygon intersects a DGGS Zone

Note

This tests whether the polygon and DGGS Zone intersect using the Hessian Normal Form to describe a plane.

Reference: Weisstein

Parameters:

• poly (array-like, required) – array/list of points defining the vertices of the polygon in ECEF XYZ Coordinates

• dggs_cell (Class-Object, required) – The DGGS_Cell class object instance for the DGGS Zone of interest.

• dggs_type (str, optional, default='TerraNexus') – The Type of DGGS Instance.

Returns:

inside_flag – Flag to indicate whether or not the polygon intersects with the DGGS Zone.

Return type:

boolean

project_pnt2surface(pnt, a=0.0, f=-1.0)

Note

Function Details

Function to project a point, or points, to the surface of the ellipsoid. This is an iterative process to refine the point coordinates (in Geodetic Latitude, Longitude, Height coordinates) until the height is within 1 millimetre from the computed ellipsoidal surface.

Parameters:

• pnt (list-like, required) – The point, or points, to be converted from Geodetic Longitude, Latitude, Height Coordinates to ECEF XYZ Coordinates

• a (float, optional, default=6378137.0) –

  The semi-major ellipsoidal radius of the Earth Model. For ellipsoids of rotation (like the Earth Model) this is equivalent to the Equatorial Radius from the centre of the Earth.

  ..note

  if "a" is omitted or set to 0.0 then the Class instance of "a" (i.e. `self.a`) will be used
  

• f (float, optional, default=-1.0) –

  The Earth Flattening Ratio. The Ratio of flattening of the polar axes verses the equatorial axes of the Earth Model to properly define the ellipsoid of rotation.

  ..note

  if "f" is omitted or set to -1.0 then the Class instance of "f" (i.e. `self.f`) will be used
  

Returns:

convertedData – A numpy array containing the converted point, or points.

Return type:

array-like

read_df_from_mongo(db, tablename, query={}, skip_n=0, limit_n=0, no_id=True)

Note

Function Details

Utility function to read data from a MongoDB Database Collection and return as a Pandas DataFrame object.

Parameters:

• db (str, required) – The name of the MongoDB Database to return data from

• tablename (str, required) – MongoDB Database Collection name to return data from

• query (JSON, optional, default={}) – The query to send to the MongoDB Database Collection.

• skip_n (int, optional, default=0) – The number of rows of data to skip.

• limit_n (int, optional, default=0) – The limit of the number of rows of data to be returned

• no_id (boolean, optional, default=True) – Flag to select whether, or not, to remove the internal MongoDB “_id” field from the data resultset.

Returns:

df – A Pandas DataFrame containing the requested data.

Return type:

Pandas DataFrame

refine_cell(refine_cell_params)

Note

Function Details

Function to refine a TerraNexus DGGS parent Zone into it’s Child Zones.

Parameters:

refine_cell_params (JSON, required) –

parameter dictionary/JSON object that drives this function. The structure of this JSON object is as follows:

refine_cell_params = {'id':                    <zoneId>,
                      'refinement_ratio':      <refinement_ratio>,
                      'parent_cell_centroid':  [x, y, z],
                      'parent_cell_verts':    [[x1, y1, z1],
                                               [x2, y2, z2],
                                               ...,
                                               [xn, yn, zn]],
                      'a0':                    <semi-major radius>,
                      'cell_type':             <cell_type>,
                      'base_polyhedron':       <base_polyhedron>}

Returns:

child_cell_dict – The Child Zone Details containing the vertices of the Child Zones of the specified Parent Zone.

Return type:

JSON

repair_TerraNexus_Admin()

Note

Function Details

Utility function to Initialise a MongoDB Database collection

Parameters:

• db (str, required) – The name of the MongoDB Database to initialise

• tablename (str, required) – MongoDB Database Collection name to initialise

• shard_db (boolean, optional, default=True) – Flag to select whether, or not, to create a sharded database across the MongoDB Cluster Infrastructure.

• capped (boolean, optional, default=True) – Flag to select whether, or not, to cap the size of this MongoDB Database Collection.

• capped_size (int, optional, default=10737418240) – Maximum disk size limit for this MongoDB database collection in bytes.

• index_spec (array-like, optional, default=[]) – Array/List that contains the specification of which database fields will be indexed and in what direction (ASCENDING or DESCENDING)

rotate_vector(rot_axis, rot_theta, input_vector)

Note

Function Details

Utility Function to rotate a vector about an axis.

Parameters:

• rot_axis (array-like, required) – The rotation axis defined in vector notation.

• rot_theta (float, required) – The angle to rotate around the “rotation_axis” in radians

• input_vector (array-like) – Input vector to rotate in ECEF (XYZ) coordinates

Returns:

rotated_vector – The rotated representation of the input vector.

Return type:

array-like

send_email(text_content, html_content='', subject='<subject>', to_email='webmaster@pangaeainnovations.com', from_email='webmaster@pangaeainnovations.com')

Note

Function Details

Utility function to send emails from the TerraNexus Web Application Server.

Parameters:

• text_content (str, required) – The plain text encoded message body of the email

• html_content (str, optional, default='') – The html encoded message body of the email

• subject (str, optional, default='<subject>') – The Subject of the email

• to_email (str, optional, default=’webmaster@pangaeainnovations.com’) – The email to send the message to.

• from_email (str, optional, default=’webmaster@pangaeainnovations.com’) – The`From` email address to indicate who this email was sent by.

simplex_volume(vertices=None, sides=None)

Note

Function Details

Utility function to Return the volume of the simplex with given vertices or sides.

If vertices are given they must be in a NumPy array with shape (N+1, N): the position vectors of the N+1 vertices in N dimensions. If the sides are given, they must be the compressed pairwise distance matrix as returned from scipy.spatial.distance.pdist.

Raises a ValueError if the vertices do not form a simplex (for example, because they are coplanar, colinear or coincident).

Warning

this algorithm has not been tested for numerical stability.

Parameters:

• vertices (array-like, optional, default=None) – Array/List of simplex vertices.

• sides (array-like, optional, default=None) – Array/List of simplex sides.

Returns:

simplex_volume_info – A tuple containing the simplex_volume and circumradius as items.

Return type:

tuple

splitDataFrameIntoSmaller(df, chunkSize=10000)

Note

Function Details

Internal utility function to split a large Pandas DataFrame into chunks.

Parameters:

• df (Pandas DataFrame, required) – The Pandas DataFrame object to split

• chunk_size (int, optional, default=10000) – Chunk size to use to split the Pandas DataFrame into smaller, more managable chunks.

Returns:

listOfDf – The list of “split” Pandas DataFrames that can now be more easily worked on in memory.

Return type:

list

store_df_to_mongo(db='test_db', tablename='test_table', df=[], remove_duplicates=True, duplicate_filter=[])

Note

Function Details

Utility function to store data to a MongoDB Database Collection

Parameters:

• db (str, required) – The name of the MongoDB Database to return data from

• tablename (str, required) – MongoDB Database Collection name to return data from

• df (Pandas DataFrame, required) – The Pandas DataFrame object to send to the MongoDB Database Collection.

• remove_duplicates (boolean, optional, default=True) – Flag to select whether, or not, to remove duplicate data records.

• duplicate_filter (array-like, optional, default=[]) – The Array/List of database collection fieldnames to use as a filter.

update_df_to_mongo(db='test_db', tablename='test_table', query={}, fields_to_update={})

Note

Function Details

Utility function to update data stored in a MongoDB Database Collection

Parameters:

• db (str, required) – The name of the MongoDB Database to return data from

• tablename (str, required) – MongoDB Database Collection name to return data from

• df (Pandas DataFrame, required) – The Pandas DataFrame object to send to the MongoDB Database Collection.

• query (JSON, optional, default={}) – The query to send to the MongoDB Database Collection.

• skip_n (int, optional, default=0) – The number of rows of data to skip.

• limit_n (int, optional, default=0) – The limit of the number of rows of data to be returned

• no_id (boolean, optional, default=True) – Flag to select whether, or not, to remove the internal MongoDB “_id” field from the data resultset.

update_nested_dictionary(dict_to_update, update_value, *keys)

Note

Function Details

Internal function to update a “Nested” Dictionary or JSON object.

Parameters:

• dict_to_update (JSON, required) – The Dictionary or JSON object to update

• update_value (object, required) – The item value to update the provided Dictionary “keys” with.

• keys (str, required) – The keys of the Dictionary or JSON object to update with the “update_value”

Returns:

updatedDictionary – The updated Dictionary or JSON object

Return type:

JSON

utm_to_ll(pnt, proj_spec='epsg:28355')

src.util.dggs_util._connect_mongo(host, port, username, password)

Note

Function Details

A backend utility for making a connection to the MongoDB Cluster.

util.logging_middleware module

class src.util.logging_middleware.SlackExceptionHandler(level=0)

Bases: AdminEmailHandler

emit(record, *args, **kwargs)

Do whatever it takes to actually log the specified logging record.

This version is intended to be implemented by subclasses and so raises a NotImplementedError.

util.ui_tools module

Created on 1 Apr 2015

@author: Dr. Matthew B.J. Purss

class src.util.ui_tools.CheckWidgetName

Bases: object

classdocs

check(parentWidget, widgetName, hideFlag=False, returnWidget=False)

class src.util.ui_tools.DBManager

Bases: object

_connect_mongo(host, port, username, password)

A util for making a connection to mongo

class src.util.ui_tools.DataManager(mprDict)

Bases: object

clear_data(key)

fw_get_dggsTiles(group_id)

get_all_data(datafields, fw_flag=False)

get_group_data(group_id, status_progressBar=None, levelling_flag=False, datafields=None, levelling_pass=1, fw_flag=False)

get_spectra(group_id, status_progressBar=None, fw_flag=False)

get_tile_data(datafields, resolution=0)

get_tile_data_error(errorMsg)

get_tile_data_success(result)

launch_tasks(fw)

save_group_data(group_id, data_df, levelling_pass=1, datafields_to_drop=None, fw_flag=False)

class src.util.ui_tools.LUT

Bases: object

ColourMap class to load pre-defined colour LUT and configure them for use with pyqtgraph widgets

build_colourMapDict()

get_all_ColourMaps()

get_colourMap(cm_name)

class src.util.ui_tools.LevellingManager(delta_stats_file)

Bases: object

getDeltaStats(tile_id='')

initialiseDeltaStats()

saveDeltaStats(updated_deltaStatsDict, tile_id='')

class src.util.ui_tools.LogManager(logName='fw_geophysicsTools', logPath='logfile.log', loggingLevel_file=10, loggingLevel_console=20, fw_flag=False)

Bases: object

getLogger()

class src.util.ui_tools.MongoFormatter(fmt=None, datefmt=None, style='%', validate=True, *, defaults=None)

Bases: Formatter

format(record)

Format exception object as a string

class src.util.ui_tools.MongoHandler(level=0, host='localhost', port=27017, username='root', password='password', database='db', collection='log', capped=True, size=100000, drop=False)

Bases: Handler

A logging handler that will record messages to a (optionally capped) MongoDB collection.

>>> connection = pymongo.Connection()
>>> collection = connection.db.log
>>> logger = logging.getLogger("mongotest")
>>> logger.addHandler(MongoHandler(drop=True))
>>> logger.error("Hello, world!")
>>> collection.find_one()['message']
u'Hello, world!'

_connect_mongo(host, port, username, password)

A util for making a connection to mongo

emit(record)

Do whatever it takes to actually log the specified logging record.

This version is intended to be implemented by subclasses and so raises a NotImplementedError.

class src.util.ui_tools.ProjectManager(projectFile='')

Bases: object

openProject()

save_data(updated_mprDict)

class src.util.ui_tools.RHEALPixDGGSExt(parent=None)

Bases: RHEALPixDGGS

cells_centroids_from_region(resolution, ul, dr, plane=True)

If plane = True, then return a list of lists of resolution resolution cells that cover the axis-aligned rectangle whose upper left and lower right vertices are the points ul and dr, respectively. In the output, sort each sublist of cells from left to right (in the planar DGGS) and sort the sublists from top to bottom.

If plane = False, then return a list of lists of resolution resolution cells that cover the longitude-latitude aligned ellipsoidal quadrangle whose northwest and southeast vertices are the points ul and dr, respectively. Defunct quads with ul = (stuff, pi/2) or dr = (stuff, -pi/2) also work (and rely on the fact that the north and south pole can both be specified by infinitely many longitudes).

To specify an ellipsoidal cap region, set ul = (-pi, pi/2) and dr = (-pi, phi) for a northern cap from latitudes pi/2 to phi, or set ul = (-pi, phi) and dr = (-pi, -pi/2) for a southern cap from latitudes phi to -pi/2. (As usual, if self.ellipsoid.radians = False, then use degrees instead of radians when specifying ul and dr.)

In the output, sort each sublist of cells from west to east (in the ellipsoidal DGGS) and sort the sublists from north to south.

Return the empty list if if ul[0] > dr[0] or ul[1] < dr[1].

NOTE:

If plane = True, then the resulting list is a matrix, that is, each sublist has the same length. This is not necessarily so if plane = False; see the examples below.

EXAMPLES:

>>> rdggs = WGS84_003_RADIANS
>>> R_A = rdggs.ellipsoid.R_A
>>> ul = R_A*array((-0.1, pi/4))
>>> dr = R_A*array((0.1, -pi/4))  # Rectangle
>>> M = rdggs.cells_from_region(1, ul, dr)
>>> for row in M:
...     print([str(cell) for cell in row])
['P2', 'Q0']
['P5', 'Q3']
['P8', 'Q6']

>>> ul = (0, pi/3)
>>> dr = (pi/2, 0)  # Quad
>>> M = rdggs.cells_from_region(1, ul, dr, plane=False)
>>> for row in M:
...     print([str(cell) for cell in row])
['N2', 'N1', 'N0']
['Q0', 'Q1', 'Q2', 'R0']
['Q3', 'Q4', 'Q5', 'R3']

>>> ul = (0, -pi/6)
>>> dr = (pi/2, -pi/2)  # Defunct quad / lune segment
>>> M = rdggs.cells_from_region(1, ul, dr, plane=False)
>>> for row in M:
...     print([str(cell) for cell in row])
['Q6', 'Q7', 'Q8', 'R6']
['S8', 'S7', 'S6']
['S4']

>>> ul = (-pi, -pi/5)
>>> dr = (-pi, -pi/2)  # Cap
>>> M = rdggs.cells_from_region(1, ul, dr, plane=False)
>>> for row in M:
...     print([str(cell) for cell in row])
['O6', 'O7', 'O8', 'P6', 'P7', 'P8', 'Q6', 'Q7', 'Q8', 'R6', 'R7', 'R8']
['S0', 'S1', 'S2', 'S5', 'S8', 'S7', 'S6', 'S3']
['S4']

class src.util.ui_tools.RepeatedTimer(interval, function, *args, **kwargs)

Bases: object

custom class to call a python process repeatedly at a set time interval.

Example Usage: from time import sleep

def hello(name):

print “Hello %s!” % name

print “starting…” rt = RepeatedTimer(1, hello, “World”) # it auto-starts, no need of rt.start()

try:

sleep(5) # your long-running job goes here…

finally:

rt.stop() # better in a try/finally block to make sure the program ends!

start()

stop()

class src.util.ui_tools.SafeHDFStore(hdf5_file, mode='r', complevel=9, complib='blosc', fletcher32=True, probe_interval=0.001)

Bases: HDFStore

util.slack_logger module

Note

Module Details Created on 13 July 2022

@author: Dr. Matthew B.J. Purss

You have two options for sending data to the Webhook URL above:

• Send a JSON string as the payload parameter in a POST request

• Send a JSON string as the body of a POST request

For a simple message, your JSON payload could contain a text property at minimum. This is the text that will be posted to the channel.

payload={"text": "This is a line of text in a channel.\nAnd this is another line of text."}

To create a link in your text, enclose the URL in <> angle brackets. For example: payload=”text”: “” will post a clickable link to https://slack.com. To display hyperlinked text instead of the actual URL, use the pipe character, as shown in this example:

payload={"text": "A very important thing has occurred! <https://alert-system.com/alerts/1234|Click here> for details!"}

You can customize the name and icon of your Incoming Webhook in the Integration Settings section below. However, you can override the displayed name by sending

"username": "new-bot-name"

in your JSON payload. You can also override the bot icon either with

"icon_url": "https://slack.com/img/icons/app-57.png"

or

"icon_emoji": ":ghost:".

Note

emoji codes are referenced between “:” characters at the beginning and end of the label. see: https://www.webfx.com/tools/emoji-cheat-sheet/ for a list of common emoji’s.

Incoming webhooks have a default channel, but it can be overridden in your JSON payload. A public channel can be specified with

"channel": "#other-channel",

and a Direct Message with

"channel": "@username".

Putting this all together, here is a sample curl request for posting to a channel.

curl -X POST --data-urlencode
    "payload={
        "channel": "#terranexus-dev-errors",
        "username": "webhookbot",
        "text": "This is posted to #terranexus-dev-errors and
                    comes from a bot named webhookbot.",
        "icon_emoji": ":ghost:"}"
        https://hooks.slack.com/services/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

class src.util.slack_logger.SlackInterface(icon_url='', icon_emoji='', channel='', username='')

Bases: object

Note

Class Details

SlackInterface Class object. This wraps the details of formating and sending messages to Slack from TerraNexus.

_send(payload)

Note

Function Details

Internal function to execute the http POST call to post the message to Slack.

Parameters:

payload (json object) – Slack Message Payload.

Note

The Pangaea Innovations Slack Workspace url is called from the “SLACK_WEBHOOK_URL” Environment Variable. This variable is set on TerraNexus deployment by the TerraNexus App Config Map Kubernetes configuration file.

post_error(err_title, err_fields, err_message)

Note

Function Details

Function to prepare Exception/Error messages to be posted to the Pangaea Innovations Slack Workspace.

Parameters:

• err_title (str) – Headline title for the slack error message.

• err_fields (array-like) – error fields to add to the message. These are typically input parameters and other information related to the python function that threw the exception.

• err_message (str) – The exception Traceback information.

post_message(message)