ripple1d.utils package#

Submodules#

ripple1d.utils.dg_utils module#

Utils for working with raster data.

ripple1d.utils.dg_utils.bbox_to_polygon(bbox)#

Convert a bounding box to a Shapely Polygon.

Return type:

Polygon

Parameters:

bbox (The bounding box to convert. It should be a sequence of four numbers: (min_x, min_y, max_x, max_y).)

Returns:

shapely.Polygon: The Shapely Polygon representing the bounding box. The Polygon is a rectangle with the lower

left corner at (min_x, min_y) and the upper right corner at (max_x, max_y).

ripple1d.utils.dg_utils.clip_raster(src_path, dst_path, mask_polygon, vertical_units)#

Clip a raster file to a polygon and save the result to a new file.

ripple1d.utils.dg_utils.convert_units(dst_crs, resolution, resolution_units)#

Convert resolution to match the units of the destination crs.

Return type:

float

ripple1d.utils.dg_utils.get_raster_bounds(raster_file)#

Retrieve the geographic bounds of a raster file and returns them in the WGS 84 (EPSG:4326) coordinate reference system.

Return type:

Tuple[float, float, float, float]

Parameters:

(str) (raster_file)

Returns:

Tuple[float, float, float, float]: The geographic bounds of the raster file in the WGS 84 (EPSG:4326) coordinate reference system. The bounds are returned as a tuple of four floats: (west, south, east, north).

ripple1d.utils.dg_utils.get_raster_metadata(raster_file)#

Retrieve the metadata of a raster file.

Return type:

dict

Parameters:

(str) (raster_file)

Returns:

dict: The metadata of the raster file. The metadata is returned as a dictionary where the keys are the names of the metadata items and the values are the values of the metadata items.

ripple1d.utils.dg_utils.get_terrain_exe_path(ras_ver)#

Return Windows path to RasProcess.exe exposing CreateTerrain subroutine, compatible with provided RAS version.

Return type:

str

ripple1d.utils.dg_utils.get_unit_name(crs)#

Get units from a crs object.

ripple1d.utils.dg_utils.reproject_raster(src_path, dest_path, dst_crs, resolution=None, resolution_units=None, COMPRESS='DEFLATE', PREDICTOR='3', num_threads=4, tiled=False, blocksize=512)#

Reproject/resample raster.

ripple1d.utils.gpkg_utils module#

Utils for working with geopackages.

ripple1d.utils.gpkg_utils.create_geom_item(item_id, bbox, footprint, properties)#

Create a PySTAC Item for a gpkg file stored in an AWS S3 bucket.

Return type:

Item

Parameters:

  • (str) (item_id)

  • (shapely.Polygon) (bbox)

  • (list[float]) (footprint)

Returns:

pystac.Item: The PySTAC Item representing the gpkg file.

ripple1d.utils.gpkg_utils.create_thumbnail_from_gpkg(gdfs)#

Create a figure displaying the geopandas dataframe provided in the gdfs dictionary.

Return type:

Figure

Parameters:

(dict) (- gdf)

ripple1d.utils.gpkg_utils.find_hash(item_metadata, asset_file)#

Extract the hash value for a given asset file from a metadata dictionary.

This function searches through a metadata dictionary for an asset file based on the file’s extension. It then extracts and returns the hash value associated with that file extension from the metadata.

Return type:

dict

Parameters:

  • (dict) (item_metadata)

  • (str) (asset_file)

Returns:

hash_dict (Dict): A dictionary with a single key-value pair where the value is the hash for the asset file. Returns an empty dictionary if no hash is found for the file extension.

ripple1d.utils.gpkg_utils.get_asset_info(asset_key, rms, bucket=None)#

Generate information for an asset based on its file extension.

Return type:

dict

Parameters:

  • (str) (bucket)

  • (str)

  • (str)

Returns:

dict: A dictionary with the roles, the description, and the title of the asset.

ripple1d.utils.gpkg_utils.get_river_miles(river_gdf)#

Compute the total length of the river centerlines in miles.

ripple1d.utils.gpkg_utils.gpkg_to_geodataframe(gpkg_s3_uri)#

Convert a local geopackage file to a GeoDataFrame.

Return type:

dict

Parameters:

(str) (gpkg_key)

Returns:

gpkg_gdf (dict): dictionary of GeoDataFrame.

ripple1d.utils.gpkg_utils.remove_hash_from_metadata(item_metadata)#

Remove “Hash” from each key (if it exists) in metedata dictionary.

Return type:

dict

Parameters:

(Dict) (item_metadata)

Returns:

no_hash_metadata (Dict): New metadata dictionary without hash info.

ripple1d.utils.gpkg_utils.reproject(gdfs, crs=4326)#

Reproject a gdf to a new CRS.

Return type:

dict

ripple1d.utils.gpkg_utils.write_thumbnail_to_s3(fig, png_s3_key, bucket, s3_client)#

Write a PNG thumbnail to AWS S3.

Parameters:

  • (plt.Figure) (- fig) –

    • png_s3_key (str): The S3 path where the generated PNG thumbnail is to be stored.

  • (str) (- bucket)

  • s3_client (-)

ripple1d.utils.ripple_utils module#

General utils used by ripple1d.

ripple1d.utils.ripple_utils.assert_no_mesh_error(compute_message_file, require_exists)#

Scan *.computeMsgs.txt for errors encountered. Raise RASComputeMeshError if found.

ripple1d.utils.ripple_utils.assert_no_ras_compute_error_message(compute_message_file)#

Scan *.computeMsgs.txt for errors encountered.

ripple1d.utils.ripple_utils.assert_no_ras_geometry_error(compute_message_file)#

Scan *.computeMsgs.txt for errors encountered.

ripple1d.utils.ripple_utils.assert_no_store_all_maps_error_message(compute_message_file)#

Scan *.computeMsgs.txt for errors encountered.

ripple1d.utils.ripple_utils.check_xs_direction(cross_sections, reach)#

Return only cross sections that are drawn right to left looking downstream.

ripple1d.utils.ripple_utils.clip_ras_centerline(centerline, xs, buffer_distance=0)#

Clip RAS centeline to the most upstream and downstream cross sections.

ripple1d.utils.ripple_utils.data_pairs_from_text_block(lines, width)#

Split lines at given width to get paired data string. Split the string in half and convert to tuple of floats.

Return type:

list[tuple[float]]

ripple1d.utils.ripple_utils.data_triplets_from_text_block(lines, width)#

Split lines at given width to get paired data string. Split the string in half and convert to tuple of floats.

Return type:

list[tuple[float]]

ripple1d.utils.ripple_utils.decode(df)#

Decode all string columns in a pandas DataFrame.

ripple1d.utils.ripple_utils.determine_crs_units(crs)#

Determine the units of the crs.

ripple1d.utils.ripple_utils.determine_junction_xs(xs, junction)#

Determine the cross sections that bound a junction.

Return type:

GeoDataFrame

ripple1d.utils.ripple_utils.determine_xs_order(row, junction_xs)#

Detemine what order cross sections bounding a junction should be in to produce a valid polygon.

ripple1d.utils.ripple_utils.fix_reversed_xs(xs, river)#

Check if cross sections are drawn right to left looking downstream. If not reverse them.

Return type:

GeoDataFrame

ripple1d.utils.ripple_utils.get_path(expected_path, client=None, bucket=None)#

Get the path for a file.

Return type:

str

ripple1d.utils.ripple_utils.handle_spaces(line, lines)#

Handle spaces in the line.

ripple1d.utils.ripple_utils.handle_spaces_arround_equals(line, lines)#

Handle spaces in the line.

Return type:

str

ripple1d.utils.ripple_utils.junction_hull(xs, junction)#

Compute and return the concave hull (polygon) for a juction.

Return type:

GeoDataFrame

ripple1d.utils.ripple_utils.prj_is_ras(path)#

Verify if prj is from hec-ras model.

ripple1d.utils.ripple_utils.replace_line_in_contents(lines, search_string, replacement, token='=')#

Split a line by a token and replaces the second half of the line (for the first occurence only!).

ripple1d.utils.ripple_utils.resample_vertices(stations, max_interval)#

Resample a set of stations so that no gaps are larger than max_interval.

Return type:

ndarray

ripple1d.utils.ripple_utils.search_contents(lines, search_string, token='=', expect_one=True)#

Split a line by a token and returns the second half of the line if the search_string is found in the first half.

Return type:

list[str]

ripple1d.utils.ripple_utils.text_block_from_start_end_str(start_str, end_strs, lines, additional_lines=0)#

Search for an exact match to the start_str and return all lines from there to a line that contains the end_str.

Return type:

list[str]

ripple1d.utils.ripple_utils.text_block_from_start_str_length(start_str, number_of_lines, lines)#

Search for an exact match to the start token and return a number of lines equal to number_of_lines.

Return type:

list[str]

ripple1d.utils.ripple_utils.text_block_from_start_str_to_empty_line(start_str, lines)#

Search for an exact match to the start_str and return all lines from there to the next empty line.

Return type:

list[str]

ripple1d.utils.ripple_utils.us_ds_xs(xs)#

Get most upstream and downstream cross sections.

ripple1d.utils.ripple_utils.validate_point(geom)#

Validate that point is of type Point. If Multipoint or Linestring create point from first coordinate pair.

ripple1d.utils.ripple_utils.xs_concave_hull(xs, junction=None)#

Compute and return the concave hull (polygon) for a set of cross sections (lines all facing the same direction).

Return type:

GeoDataFrame

ripple1d.utils.s3_utils module#

Utils for working with s3.

ripple1d.utils.s3_utils.check_s3_key_exists(bucket, key)#

Check if an object with the given key exists in the specified S3 bucket.

Return type:

bool

ripple1d.utils.s3_utils.copy_item_to_s3(item, s3_key, s3client)#

Copy an item to an AWS S3 bucket.

Parameters:

  • item (The item to copy. It must have a to_dict method that returns a dictionary representation of it.)

  • (str) (s3_key)

The function performs the following steps:

  1. Initializes a boto3 S3 client and splits the s3_key into the bucket name and the key.

  2. Converts the item to a dictionary, serializes it to a JSON string, and encodes it to bytes.

  3. Puts the encoded JSON string to the specified file path in the S3 bucket.

ripple1d.utils.s3_utils.extract_bucketname_and_keyname(s3path)#

Parse the provided s3:// object path and return its bucket name and key.

Return type:

tuple[str, str]

ripple1d.utils.s3_utils.get_basic_object_metadata(obj)#

Retrieve basic metadata of an AWS S3 object.

Return type:

dict

Parameters:

(ObjectSummary) (obj)

Returns:

dict: A dictionary with the size, ETag, last modified date, storage platform, region, and storage tier of the object.

ripple1d.utils.s3_utils.init_s3_resources()#

Establish a boto3 (AWS) session and return the session, S3 client, and S3 resource handles.

Return type:

tuple

ripple1d.utils.s3_utils.list_keys(s3_client, bucket, prefix, suffix='')#

List all keys in an S3 bucket with a given prefix and suffix.

Return type:

list

ripple1d.utils.s3_utils.list_keys_regex(s3_client, bucket, prefix_includes, suffix='')#

List all keys in an S3 bucket with a given prefix and suffix.

Return type:

list

ripple1d.utils.s3_utils.read_json_from_s3(bucket, key)#

Read a JSON file from an S3 bucket and return its contents as a dictionary.

Return type:

dict

ripple1d.utils.s3_utils.s3_delete_dir_recursively(s3_dir, s3_resource)#

Delete a s3:// directory and its contents recursively. OK if dir does not exist.

Return type:

None

ripple1d.utils.s3_utils.s3_get_output_s3path(s3_bucket, stac_href)#

Return the s3 path to the output directory for the given stac_href.

Return type:

str

ripple1d.utils.s3_utils.s3_get_ripple1d_status_file_key_names(stac_href, s3_bucket, s3_client)#

Return two S3 key paths, the first to a succeed sentinel file, the second t oa failure sentinel file.

This function does not check if the keys exist.

Return type:

tuple[str, str]

ripple1d.utils.s3_utils.s3_key_public_url_converter(url, dev_mode=False)#

Convert an S3 URL to an HTTPS URL and vice versa.

Return type:

str

Args:#

url (str): The URL to convert. It should be in the format ‘s3://bucket/’ or ‘https://bucket.s3.amazonaws.com/’. dev_mode (bool): A flag indicating whether the function should use the Minio endpoint for S3 URL conversion.

Return:#

:

str: The converted URL. If the input URL is an S3 URL, the function returns an HTTPS URL. If the input URL is an HTTPS URL, the function returns an S3 URL.

The function performs the following steps:

  1. Checks if the input URL is an S3 URL or an HTTPS URL.

  2. If the input URL is an S3 URL, it converts it to an HTTPS URL.

  3. If the input URL is an HTTPS URL, it converts it to an S3 URL.

ripple1d.utils.s3_utils.s3_ripple1d_status_succeed_file_exists(stac_href, s3_bucket, s3_client)#

Check if the standard ripple1d succeed sentinel file exists. If it does, return True, otherwise return False.

Return type:

bool

ripple1d.utils.s3_utils.s3_upload_dir_recursively(local_src_dir, tgt_dir, s3_client)#

Copy all files from a local directory. tgt_dir can be local or a s3:// prefix.

Return type:

None

ripple1d.utils.s3_utils.s3_upload_status_file(stac_href, s3_bucket, s3_client, e)#

Upload a status file to s3.

If e is a Python exception, then upload a ‘fail’ json file to the href’s standard output location on s3. If e is None, then upload a ‘succeed’ json file. Either file will have key “time” indicating the time that the file was uploaded. A ‘fail’ file will also have keys “err” and “traceback” containing the exception as a string and the Python traceback of the exception, respectively.

ripple1d.utils.s3_utils.split_s3_key(s3_key)#

Split an S3 key into the bucket name and the key.

Return type:

tuple[str, str]

Parameters:

(str) (s3_key)

Returns:

  • tuple: A tuple containing the bucket name and the key. If the S3 key does not contain a key, the second element

    of the tuple will be None.

  • The function performs the following steps –

    1. Removes the ‘s3://’ prefix from the S3 key.

    2. Splits the remaining string on the first ‘/’ character.

    3. Returns the first part as the bucket name and the second part as the key. If there is no ‘/’, the key will

    be None.

ripple1d.utils.s3_utils.str_from_s3(ras_text_file_path, client, bucket)#

Read a text file from s3 and return its contents as a string.

Return type:

str

ripple1d.utils.s3_utils.verify_safe_prefix(s3_key)#

TODO: discuss this with the team. Would like some safety mechanism to ensure that the S3 key is limited to certain prefixes.

Should there be some restriction where these files can be written?

ripple1d.utils.sqlite_utils module#

Utils for working with sqlite databases.

ripple1d.utils.sqlite_utils.agreement_dict_sql_prep(d, table)#

Preprocess a dict to be easily instered with insertmany().

ripple1d.utils.sqlite_utils.create_db_and_table(db_name, table_name)#

Create sqlite database and table.

ripple1d.utils.sqlite_utils.create_non_spatial_table(gpkg_path, metadata)#

Create the metadata table in the geopackage.

Return type:

None

ripple1d.utils.sqlite_utils.create_terrain_agreement_db(out_path)#

Initialize agreement database.

ripple1d.utils.sqlite_utils.export_terrain_agreement_metrics_to_db(out_path, metrics)#

Export terrain agreement dict to a sqlite database.

ripple1d.utils.sqlite_utils.insert_data(db_name, table_name, data, plan_suffix, missing_grids, boundary_condition)#

Insert data into the sqlite database.

ripple1d.utils.sqlite_utils.parse_stage_flow(wses)#

Parse flow and control by stage from profile names.

Return type:

DataFrame

ripple1d.utils.sqlite_utils.rating_curves_to_sqlite(rm, plan_name, plan_suffix, nwm_id, missing_grids, database_path, table_name)#

Export rating curves to sqlite.

ripple1d.utils.sqlite_utils.zero_depth_to_sqlite(rm, plan_name, plan_suffix, nwm_id, missing_grids, database_path, table_name)#

Export zero depth (normal depth) results to sqlite.

ripple1d.utils.stac_utils module#

Utils for working with STAC items/catalogs.

ripple1d.utils.stac_utils.collection_exists(endpoint, collection_id)#

Check if a collection exists in a STAC API.

ripple1d.utils.stac_utils.create_collection(models, id, description=None, title=None)#

Create a STAC collection from a list of STAC items.

Return type:

Collection

ripple1d.utils.stac_utils.delete_collection(endpoint, collection_id, headers)#

Upsert a collection to a STAC API.

ripple1d.utils.stac_utils.download_model_from_stac_item(stac_item, ras_directory, client, bucket)#

Download HEC-RAS model from stac href. Return the terrain_name.

Return type:

str

ripple1d.utils.stac_utils.key_to_uri(key, bucket)#

Convert a key to a uri.

Return type:

str

ripple1d.utils.stac_utils.upsert_collection(endpoint, collection, headers)#

Upsert a collection to a STAC API.

ripple1d.utils.stac_utils.upsert_item(endpoint, collection_id, item, headers)#

Upsert an item to a STAC API.

ripple1d.utils.stac_utils.uri_to_key(href, bucket)#

Convert a uri to a key.

Return type:

str

Module contents#

Initialize utils.

On this page
Show Source