`amescap.Script_utils`

Script_utils contains internal functions for processing netCDF files. These functions can be used on their own outside of CAP if they are imported as a module:

from /u/path/Script_utils import MY_func

Third-party Requirements:

numpy

netCDF4

re

os

subprocess

sys

math

matplotlib.colors

Module Contents

Functions

`FV3_file_type`(fNcdf)	Return the type of the netCDF file.
`MY_func`(Ls_cont)	Returns the Mars Year.
`ak_bk_loader`(fNcdf)	Loads the ak and bk variables from a netCDF file.
`alt_FV3path`(fullpaths, alt[, test_exist])	Returns the original or fixed file for a given path.
`check_bounds`(values, min_val, max_val, dx)	Checks the bounds of a variable in a netCDF file.
`check_file_tape`(fileNcdf)	Checks whether a file exists on the disk.
`dkass_dust_cmap`()	Color map (From Courtney Batterson).
`dkass_temp_cmap`()	Color map (From Courtney Batterson).
`except_message`(debug, exception, varname, ifile[, ...])	Prints an error message if a variable is not found.
`extract_path_basename`(filename)	Returns the path and basename of a file.
`filter_vars`(fNcdf[, include_list, giveExclude])	Filters the variable names in a netCDF file for processing.
`find_fixedfile`(filename)	Finds the relevant fixed file for an average, daily, or diurn file.
`find_tod_in_diurn`(fNcdf)	Returns the variable for the local time axis in diurn files.
`get_Ncdf_path`(fNcdf)	Returns the full path for a netCDF file object.
`get_longname_unit`(fNcdf, varname)	Returns the longname and unit of a variable.
`give_permission`(filename)	Sets group file permissions for the NAS system
`hot_cold_cmap`()	Returns a color map (From Alex Kling, based on bipolar cmap).
`pretty_print_to_fv_eta`(var, varname[, nperline])	Print the `ak` or `bk` coefficients for copying to `fv_eta.f90`.
`print_fileContent`(fileNcdf)	Prints the contents of a netCDF file to the screen.
`print_varContent`(fileNcdf, list_varfull[, print_stat])	Print variable contents from a variable in a netCDF file.
`progress`(k, Nmax)	Displays a progress bar to monitor heavy calculations.
`read_variable_dict_amescap_profile`([f_Ncdf])	Reads a variable dictionary from the `amescap_profile` file.
`regrid_Ncfile`(VAR_Ncdf, file_Nc_in, file_Nc_target)	Regrid a netCDF variable from one file structure to another.
`replace_dims`(Ncvar_dim[, vert_dim_name])	Replaces the dimensions of a variable in a netCDF file.
`reset_FV3_names`(MOD)	Resets the FV3 variable names in a netCDF file.
`rjw_cmap`()	Returns a color map (From R. John Wilson).
`section_content_amescap_profile`(section_ID)	Executes first code section in `~/.amescap_profile`.
`smart_reader`(fNcdf, var_list[, suppress_warning])	Reads a variable from a netCDF file.
`wbr_cmap`()	Returns a color map (From R. John Wilson).

Attributes

`Blue`
`Cyan`
`Green`
`Nclr`
`Purple`
`Red`
`Yellow`

FV3_file_type(fNcdf)

Return the type of the netCDF file.

Returns netCDF file type (i.e., fixed, diurn, average, daily) and the format of the Ls array areo (i.e., fixed, continuous, or diurn).

Parameters:

fNcdf (Netcdf file object) – an open Netcdf file

Returns:

The Ls array type (string, fixed, continuous, or diurn) and the netCDF file type (string fixed, diurn, average, or daily)

Return type:

tuple

Raises:

ValueError – if the file is not a netCDF file
FileNotFoundError – if the file does not exist
TypeError – if the file is not a Dataset or MFDataset
AttributeError – if the file does not have the _files or filepath attribute

MY_func(Ls_cont)

Returns the Mars Year.

Parameters:: Ls_cont (array) – solar longitude (continuous)
Returns:: the Mars year
Return type:: int
Raises:: ValueError – if Ls_cont is not in the range [0, 360)

ak_bk_loader(fNcdf)

Loads the ak and bk variables from a netCDF file.

This function will first check the current netCDF file for the ak and bk variables. If they are not found, it will search the fixed file in the same directory. If they are still not found, it will search the tiled fixed files. The function will return the ak and bk arrays.

Parameters:: fNcdf (a netCDF file object) – an open netCDF file
Returns:: the ak and bk arrays
Return type:: tuple
Raises:: ValueError – if the ak and bk variables are not found in the netCDF file, the fixed file, or the tiled fixed files

Note

This routine will look for both ak and bk. There are cases when it is convenient to load the ak, bk once when the files are first opened in MarsVars, but the ak and bk arrays may not be necessary for in the calculation as is the case for MarsVars XXXXX.atmos_average_psd.nc --add msf, which operates on a pressure interpolated (_pstd.nc) file.

alt_FV3path(fullpaths, alt, test_exist=True)

Returns the original or fixed file for a given path.

Parameters:

fullpaths (str) – full path to a file or a list of full paths to more than one file
alt (str) – type of file to return (i.e., original or fixed)
test_exist (bool, optional) – Whether file exists on the disk, defaults to True

Returns:

path to original or fixed file (e.g., /u/path/00010.atmos_average.nc or /u/path/00010.fixed.nc)

Return type:

str

Raises:

ValueError – if the file is not a netCDF file
FileNotFoundError – if the file does not exist
TypeError – if the file is not a Dataset or MFDataset
AttributeError – if the file does not have the _files or filepath attribute
OSError – if the file is not a valid path

check_bounds(values, min_val, max_val, dx)

Checks the bounds of a variable in a netCDF file.

This function checks if the values in a netCDF file are within the specified bounds. If any value is out of bounds, it will print an error message and exit the program. The function can handle both single values and arrays.

Parameters: :param values: Single value or array of values to check :type values: array-like :param min_val: Minimum allowed value :type min_val: float :param max_val: Maximum allowed value :type max_val: float :return values: The validated value(s) :rtype: array or float :raises ValueError: if values is out of bounds or if values is not

a number, array, or list

check_file_tape(fileNcdf)

Checks whether a file exists on the disk.

If on a NAS system, also checks if the file needs to be migrated from tape.

Parameters:

fileNcdf (str or file object) – full path to a netcdf file or a file object with a name attribute

Returns:

None

Raises:

ValueError – if the file is not a netCDF file
FileNotFoundError – if the file does not exist
subprocess.CalledProcessError – if the dmls command fails

dkass_dust_cmap()

Color map (From Courtney Batterson).

Returns a color map useful for dust cross-sections that highlight dust mixing ratios > 4 ppm. The color map goes from white -> yellow -> orange -> red -> purple. [Courtney Batterson, Nov 2022]

Returns:: color map
Return type:: array

dkass_temp_cmap()

Color map (From Courtney Batterson).

Returns a color map useful for highlighting the 200 K temperature level. The color map goes from black -> purple -> blue -> green -> yellow -> orange -> red. [Courtney Batterson, Nov 2022]

Returns:: color map
Return type:: array

except_message(debug, exception, varname, ifile, pre='', ext='')

Prints an error message if a variable is not found.

It also contains a special error in the case of a pre-existing variable.

Parameters:

debug (logical) – Flag for debug mode
exception (class object) – Exception from try statement
varname (string) – Name of variable causing exception
ifile (string) – Name of input file
pre (string) – Prefix to new variable
ext (string) – Extension to new variable

Returns:

None

Return type:

None

Raises:

ValueError – if debug is True, exception is not a class object or string, varname is not a string, ifile is not a string, pre is not a string, or ext is not a string

extract_path_basename(filename)

Returns the path and basename of a file.

If only the filename is provided, assume it is in the current directory.

Parameters:

filename (str) – name of the netCDF file (may include full path)

Returns:

full file path & name of file

Return type:

tuple

Raises:

ValueError – if the filename is not a string
FileNotFoundError – if the file does not exist
TypeError – if the filename is not a string
OSError – if the filename is not a valid path

Note

This routine does not confirm that the file exists. It operates on the provided input string.

filter_vars(fNcdf, include_list=None, giveExclude=False)

Filters the variable names in a netCDF file for processing.

Returns all dimensions (lon, lat, etc.), the areo variable, and any other variable listed in include_list.

Parameters:: fNcdf (netCDF file object) – an open netCDF object for a diurn, daily, or average file

:param include_list:list of variables to include (e.g., [ucomp,: vcomp], defaults to None

Parameters:

giveExclude (bool, optional) – if True, returns variables to be excluded from the file, defaults to False

Returns:

list of variable names to include in the processed file

Return type:

list

Raises:

ValueError – if the file is not a netCDF file
FileNotFoundError – if the file does not exist
TypeError – if the file is not a Dataset or MFDataset
AttributeError – if the file does not have the _files or filepath attribute
OSError – if the file is not a valid path
KeyError – if the variable is not found in the file

find_fixedfile(filename)

Finds the relevant fixed file for an average, daily, or diurn file.

[Courtney Batterson, updated by Alex Nov 29 2022]

Parameters:

filename (str) – an average, daily, or diurn netCDF file

Returns:

full path to the correspnding fixed file

Return type:

str

Raises:

ValueError – if the file is not a netCDF file
FileNotFoundError – if the file does not exist
TypeError – if the file is not a Dataset or MFDataset
AttributeError – if the file does not have the _files or filepath attribute
OSError – if the file is not a valid path

Compatible file types:

DDDDD.atmos_average.nc              -> DDDDD.fixed.nc
atmos_average.tileX.nc              -> fixed.tileX.nc
DDDDD.atmos_average_plevs.nc        -> DDDDD.fixed.nc
DDDDD.atmos_average_plevs_custom.nc -> DDDDD.fixed.nc
atmos_average.tileX_plevs.nc        -> fixed.tileX.nc
atmos_average.tileX_plevs_custom.nc -> fixed.tileX.nc
atmos_average_custom.tileX_plevs.nc -> fixed.tileX.nc

find_tod_in_diurn(fNcdf)

Returns the variable for the local time axis in diurn files.

(e.g., time_of_day_24). Original implementation by Victoria H.

Parameters:: fNcdf (netCDF file object) – a netCDF file
Returns:: the name of the time of day dimension
Return type:: str
Raises:: ValueError – if the time of day variable is not found

get_Ncdf_path(fNcdf)

Returns the full path for a netCDF file object.

Dataset and multi-file dataset (MFDataset) have different attributes for the path, hence the need for this function.

Parameters:

fNcdf (netCDF file object) – Dataset or MFDataset object

Returns:

string list for the Dataset (MFDataset)

Return type:

str(list)

Raises:

TypeError – if the file is not a Dataset or MFDataset
AttributeError – if the file does not have the _files or filepath attribute
ValueError – if the file is not a netCDF file
FileNotFoundError – if the file does not exist

get_longname_unit(fNcdf, varname)

Returns the longname and unit of a variable.

If the attributes are unavailable, returns blank strings to avoid an error.

Parameters:

fNcdf (netCDF file object) – an open netCDF file
varname (str) – variable to extract attribute from

Returns:

longname and unit attributes

Return type:

str

Raises:

ValueError – if the file is not a netCDF file
FileNotFoundError – if the file does not exist
TypeError – if the file is not a Dataset or MFDataset
AttributeError – if the file does not have the _files or filepath attribute
OSError – if the file is not a valid path
KeyError – if the variable is not found in the file

Note

Some functions in MarsVars edit the units (e.g., [kg] -> [kg/m]), therefore the empty string is 4 characters in length (” ” instead of “”) to allow for editing by editing units_txt[:-2], for example.

give_permission(filename)

Sets group file permissions for the NAS system

Parameters:

filename (str) – full path to the netCDF file

Returns:

None

Raises:

subprocess.CalledProcessError – if the setfacl command fails
FileNotFoundError – if the file is not found

hot_cold_cmap()

Returns a color map (From Alex Kling, based on bipolar cmap).

Color map goes from dark blue -> light blue -> white -> yellow -> red. Based on Matlab’s bipolar colormap. [Alex Kling, Nov 2022]

Returns:: color map
Return type:: array

pretty_print_to_fv_eta(var, varname, nperline=6)

Print the ak or bk coefficients for copying to fv_eta.f90.

The ak and bk coefficients are used to calculate the vertical coordinate transformation.

Parameters:

var (array) – ak or bk data
varname (str) – the variable name (“a” or “b”)
nperline (int, optional) – the number of elements per line, defaults to 6

Returns:

a print statement for copying into fv_eta.f90

Return type:

None

Raises:

ValueError – if varname is not “a” or “b”
ValueError – if nperline is not a positive integer
ValueError – if var is not a 1D array of length NLAY+1

print_fileContent(fileNcdf)

Prints the contents of a netCDF file to the screen.

Variables sorted by dimension.

Parameters:: fileNcdf (str) – full path to the netCDF file
Returns:: None

print_varContent(fileNcdf, list_varfull, print_stat=False)

Print variable contents from a variable in a netCDF file.

Requires a XXXXX.fixed.nc file in the current directory.

Parameters:

fileNcdf (str) – full path to a netcdf file
list_varfull (list) – list of variable names and optional slices (e.g., ["lon", "ps[:, 10, 20]"])
print_stat (bool, optional) – If True, print min, mean, and max. If False, print values. Defaults to False

Returns:

None

Raises:

ValueError – if the variable is not found in the file
FileNotFoundError – if the file is not found
Exception – if the variable is not found in the file
Exception – if the file is not found

progress(k, Nmax)

Displays a progress bar to monitor heavy calculations.

Parameters:

k (int) – current iteration of the outer loop
Nmax (int) – max iteration of the outer loop

Returns:

None

Raises:

ValueError – if k or Nmax are not integers, k > Nmax, or k < 0

read_variable_dict_amescap_profile(f_Ncdf=None)

Reads a variable dictionary from the amescap_profile file.

This function will read the variable dictionary from the amescap_profile file and return a dictionary with the variable names and dimensions. The function will also check the opened netCDF file for the variable names and dimensions.

Parameters:

f_Ncdf (File object) – An opened Netcdf file object

Returns:

MOD, a class object with the variable names and dimensions (e.g., MOD.ucomp = ‘U’ or MOD.dim_lat = ‘latitudes’)

Rtype MOD:

class object

Raises:

ValueError – if the amescap_profile file is not found or if the variable dictionary is not found
ValueError – if the variable or dimension name is not found in the netCDF file
ValueError – if the variable or dimension name is not found in the``amescap_profile``

regrid_Ncfile(VAR_Ncdf, file_Nc_in, file_Nc_target)

Regrid a netCDF variable from one file structure to another.

Requires a file with the desired file structure to mimic. [Alex Kling, May 2021]

Parameters:

VAR_Ncdf (netCDF file variable) – a netCDF variable object to regrid (e.g., f_in.variable["temp"])
file_Nc_in (netCDF file object) – an open netCDF file to source for the variable (e.g., f_in = Dataset("filename", "r"))
file_Nc_target (netCDF file object) – an open netCDF file with the desired file structure (e.g., f_out = Dataset("filename", "r"))

Returns:

the values of the variable interpolated to the target file grid.

Return type:

array

Raises:

ValueError – if the file is not a netCDF file
FileNotFoundError – if the file does not exist
TypeError – if the file is not a Dataset or MFDataset
AttributeError – if the file does not have the _files or filepath attribute
OSError – if the file is not a valid path

Note

While the KDTree interpolation can handle a 3D dataset (lon/lat/lev instead of just 2D lon/lat), the grid points in the vertical are just a few (10–100s) meters in the PBL vs a few (10-100s) kilometers in the horizontal. This results in excessive weighting in the vertical, which is why the vertical dimension is handled separately.

replace_dims(Ncvar_dim, vert_dim_name=None)

Replaces the dimensions of a variable in a netCDF file.

Updates the name of the variable dimension to match the format of the new NASA Ames Mars GCM output files.

Parameters:

Ncvar_dim (str) – netCDF variable dimensions (e.g., f_Ncdf.variables["temp"].dimensions)
vert_dim_name (str, optional) – the vertical dimension if it is ambiguous (pstd, zstd, or zagl). Defaults to None

Returns:

updated dimensions

Return type:

str

Raises:

ValueError – if Ncvar_dim is not a list
ValueError – if vert_dim_name is not a string
ValueError – if vert_dim_name is not in the list of recognized vertical dimensions

reset_FV3_names(MOD)

Resets the FV3 variable names in a netCDF file.

This function resets the model dictionary to the native FV3 variable names, e.g.:

model.dim_lat = 'latitude' > model.dim_lat = 'lat'
model.ucomp   = 'U'        > model.ucomp = 'ucomp'

Parameters:: MOD (class object) – Generated with read_variable_dict_amescap_profile()
Returns:: same object with updated names for the dimensions and

variables :rtype: class object :raises ValueError: if the MOD object is not a class object or does

not contain the expected attributes

rjw_cmap()

Returns a color map (From R. John Wilson).

Color map goes from red -> jade -> wisteria. [R. John Wilson, Nov 2022]

Returns:: color map
Return type:: array

section_content_amescap_profile(section_ID)

Executes first code section in ~/.amescap_profile.

Reads in user-defined plot & interpolation settings. [Alex Kling, Nov 2022]

Parameters:: section_ID (str) – the section to load (e.g., Pressure definitions for pstd)
Returns:: the relevant line with Python syntax
Return type:: str
Raises:: FileNotFoundError – if the file is not found

smart_reader(fNcdf, var_list, suppress_warning=False)

Reads a variable from a netCDF file.

If the variable is not found in the file, it checks for the variable in the original file (e.g., atmos_average.nc) or the fixed file (e.g., 00010.fixed.nc).

Alternative to var = fNcdf.variables["var"][:] for handling processed files.

Parameters:

fNcdf (netCDF file object) – an open netCDF file
var_list (_type_) – a variable or list of variables (e.g., areo or [pk, bk, areo])
suppress_warning (bool, optional) – suppress debug statement. Useful if a variable is not expected to be in the file anyway. Defaults to False

Returns:

variable content (single or values to unpack)

Return type:

list

Raises:

ValueError – if the file is not a netCDF file
FileNotFoundError – if the file does not exist
TypeError – if the file is not a Dataset or MFDataset
AttributeError – if the file does not have the _files or filepath attribute
OSError – if the file is not a valid path

Example:

from netCDF4 import Dataset

fNcdf = Dataset("/u/akling/FV3/00668.atmos_average_pstd.nc", "r")

# Approach using var = fNcdf.variables["var"][:]
ucomp = fNcdf.variables["ucomp"][:]
# New approach that checks for matching average/daily & fixed
vcomp = smart_reader(fNcdf, "vcomp")

# This will pull "areo" from an original file if it is not
# available in the interpolated file. If pk and bk are also not
# in the average file, it will check for them in the fixed file.
pk, bk, areo = smart_reader(fNcdf, ["pk", "bk", "areo"])

Note

Only the variable content is returned, not attributes

wbr_cmap()

Returns a color map (From R. John Wilson).

Color map goes from white -> blue -> green -> yellow -> red [R. John Wilson, Nov 2022]

Returns:: color map
Return type:: array

Blue = '\x1b[94m'

Cyan = '\x1b[96m'

Green = '\x1b[92m'

Nclr = '\x1b[00m'

Purple = '\x1b[95m'

Red = '\x1b[91m'

Yellow = '\x1b[93m'

amescap.Script_utils

Module Contents

Functions

Attributes

`amescap.Script_utils`