bin.MarsFiles

The MarsFiles executable has functions for manipulating entire files. The capabilities include time-shifting, binning, and regridding data, as well as band pass filtering, tide analysis, zonal averaging, and extracting variables from files.

The executable requires:

  • [input_file] The file for manipulation

and optionally accepts:

  • [-bin, --bin_files] Produce MGCM ‘fixed’, ‘diurn’, ‘average’ and ‘daily’ files from Legacy output

  • [-c, --concatenate] Combine sequential files of the same type into one file

  • [-split, --split] Split file along a specified dimension or extracts slice at one point along the dim

  • [-t, --time_shift] Apply a time-shift to ‘diurn’ files

  • [-ba, --bin_average] Bin MGCM ‘daily’ files like ‘average’ files

  • [-bd, --bin_diurn] Bin MGCM ‘daily’ files like ‘diurn’ files

  • [-hpt, --high_pass_temporal] Temporal filter: high-pass

  • [-lpt, --low_pass_temporal] Temporal filter: low-pass

  • [-bpt, --band_pass_temporal] Temporal filter: band-pass

  • [-trend, --add_trend] Return amplitudes only (use with temporal filters)

  • [-hps, --high_pass_spatial] Spatial filter: high-pass

  • [-lps, --low_pass_spatial] Spatial filter: low-pass

  • [-bps, --band_pass_spatial] Spatial filter: band-pass

  • [-tide, --tide_decomp] Extract diurnal tide and its harmonics

  • [-recon, --reconstruct] Reconstruct the first N harmonics

  • [-norm, --normalize] Provide -tide result in % amplitude

  • [-prop, --prop_tides] Extract propagating tide harmonics

  • [-regrid, --regrid_XY_to_match] Regrid a target file to match a source file

  • [-zavg, --zonal_average] Zonally average all variables in a file

  • [-incl, --include] Only include specific variables in a calculation

  • [-ext, --extension] Create a new file with a unique extension instead of modifying the current file

Third-party Requirements:

  • sys

  • argparse

  • os

  • warnings

  • re

  • numpy

  • netCDF4

  • shutil

  • functools

  • traceback

Module Contents

Classes

ExtAction

Custom action for argparse to handle file extensions.

ExtArgumentParser

Custom ArgumentParser that handles file extensions for output files.

Functions

change_vname_longname_unit(vname, longname_txt, units_txt)

Update variable name, longname, and units. This is

concatenate_files(file_list, full_file_list)

Concatenates sequential output files in chronological order.

debug_wrapper(func)

A decorator that wraps a function with error handling

do_avg_vars(histfile, newf, avgtime, avgtod[, bin_period])

Performs a time average over all fields in a file.

ls2sol_1year(Ls_deg[, offset, round10])

Returns a sol number from the solar longitude.

main()

Main entry point for MarsFiles data processing utility.

make_FV3_files(fpath, typelistfv3[, renameFV3])

Make MGCM-like average, daily, and diurn files.

process_time_shift(file_list)

Converts diurn files to local time.

replace_at_index(tuple_dims, idx, new_name)

Replaces the dimension at the given index with a new name.

replace_dims(dims, todflag)

Replaces dimensions with MGCM-like names. Removes time_of_day.

split_files(file_list, split_dim)

Extracts variables in the file along the specified dimension.

Attributes

all_args

args

debug

exit_code

out_ext

out_ext

parser

class ExtAction(*args, ext_content=None, parser=None, **kwargs)

Bases: argparse.Action

Custom action for argparse to handle file extensions.

This action is used to add an extension to the output file.

Parameters:

  • ext_content (str) – The content to be added to the file name.

  • parser (argparse.ArgumentParser) – The parser instance to which this action belongs.

  • args (tuple) – Additional positional arguments.

  • kwargs (dict) – Additional keyword arguments.

  • ext_content – The content to be added to the file name

Returns:

None

Return type:

None

Raises:

  • ValueError – If ext_content is not provided.

  • TypeError – If parser is not an instance of argparse.ArgumentParser.

  • Exception – If an error occurs during the action.

  • AttributeError – If the parser does not have the specified attribute.

  • ImportError – If the parser cannot be imported.

  • RuntimeError – If the parser cannot be run.

  • KeyError – If the parser does not have the specified key.

  • IndexError – If the parser does not have the specified index.

  • IOError – If the parser cannot be opened.

  • OSError – If the parser cannot be accessed.

  • EOFError – If the parser cannot be read.

  • MemoryError – If the parser cannot be allocated.

  • OverflowError – If the parser cannot be overflowed.

__call__(parser, namespace, values, option_string=None)
__repr__()

Return repr(self).

format_usage()
class ExtArgumentParser(prog=None, usage=None, description=None, epilog=None, parents=[], formatter_class=HelpFormatter, prefix_chars='-', fromfile_prefix_chars=None, argument_default=None, conflict_handler='error', add_help=True, allow_abbrev=True, exit_on_error=True)

Bases: argparse.ArgumentParser

Custom ArgumentParser that handles file extensions for output files.

This class extends the argparse.ArgumentParser to add functionality for handling file extensions in the command-line arguments.

Parameters:

  • args (tuple) – Additional positional arguments.

  • kwargs (dict) – Additional keyword arguments.

  • ext_content (str) – The content to be added to the file name.

  • parser (argparse.ArgumentParser) – The parser instance to which this action belongs.

Returns:

None

Return type:

None

Raises:

  • ValueError – If ext_content is not provided.

  • TypeError – If parser is not an instance of argparse.ArgumentParser.

  • Exception – If an error occurs during the action.

  • AttributeError – If the parser does not have the specified attribute.

  • ImportError – If the parser cannot be imported.

  • RuntimeError – If the parser cannot be run.

  • KeyError – If the parser does not have the specified key.

  • IndexError – If the parser does not have the specified index.

  • IOError – If the parser cannot be opened.

  • OSError – If the parser cannot be accessed.

  • EOFError – If the parser cannot be read.

  • MemoryError – If the parser cannot be allocated.

  • OverflowError – If the parser cannot be overflowed.

__repr__()

Return repr(self).

add_argument(*args, **kwargs)

add_argument(dest, …, name=value, …) add_argument(option_string, option_string, …, name=value, …)

add_argument_group(*args, **kwargs)
add_mutually_exclusive_group(**kwargs)
add_subparsers(**kwargs)
convert_arg_line_to_args(arg_line)
error(message)

error(message: string)

Prints a usage message incorporating the message to stderr and exits.

If you override this in a subclass, it should not return – it should either exit or raise an exception.

exit(status=0, message=None)
format_help()
format_usage()
get_default(dest)
parse_args(*args, **kwargs)
parse_intermixed_args(args=None, namespace=None)
parse_known_args(args=None, namespace=None)
parse_known_intermixed_args(args=None, namespace=None)
print_help(file=None)
print_usage(file=None)
register(registry_name, value, object)
set_defaults(**kwargs)
change_vname_longname_unit(vname, longname_txt, units_txt)

Update variable name, longname, and units. This is designed to work specifically with LegacyCGM.nc files.

Parameters:

  • vname (str) – variable name

  • longname_txt (str) – variable description

  • units_txt (str) – variable units

Returns:

variable name and corresponding description and unit (e.g. vname = "ps")

Return type:

tuple

Raises:

  • KeyError – if the required variables are not found

  • ValueError – if the required dimensions are not found

  • AttributeError – if the required attributes are not found

Note::

The diurn file is created by binning the Legacy files. The average and daily files are created by averaging over the diurn file.

concatenate_files(file_list, full_file_list)

Concatenates sequential output files in chronological order.

Parameters:

  • file_list (list) – list of file names

  • full_file_list (list) – list of file names and full paths

Returns:

None

Return type:

None

Raises:

  • OSError – If the file cannot be removed.

  • IOError – If the file cannot be moved.

  • Exception – If the file cannot be opened.

  • ValueError – If the file cannot be accessed.

  • TypeError – If the file is not of the correct type.

  • IndexError – If the file does not have the correct index.

  • KeyError – If the file does not have the correct key.

debug_wrapper(func)

A decorator that wraps a function with error handling based on the –debug flag.

If the –debug flag is set, it prints the full traceback of any exception that occurs. Otherwise, it prints a simplified error message.

Parameters:

func (function) – The function to wrap.

Returns:

The wrapped function.

Return type:

function

Raises:

  • Exception – If an error occurs during the function call.

  • TypeError – If the function is not callable.

  • ValueError – If the function is not found.

  • NameError – If the function is not defined.

  • AttributeError – If the function does not have the specified attribute.

  • ImportError – If the function cannot be imported.

  • RuntimeError – If the function cannot be run.

  • KeyError – If the function does not have the specified key.

  • IndexError – If the function does not have the specified index.

  • IOError – If the function cannot be opened.

  • OSError – If the function cannot be accessed.

  • EOFError – If the function cannot be read.

  • MemoryError – If the function cannot be allocated.

  • OverflowError – If the function cannot be overflowed.

  • ZeroDivisionError – If the function cannot be divided by zero.

  • StopIteration – If the function cannot be stopped.

  • KeyboardInterrupt – If the function cannot be interrupted.

  • SystemExit – If the function cannot be exited.

  • AssertionError – If the function cannot be asserted.

do_avg_vars(histfile, newf, avgtime, avgtod, bin_period=5)

Performs a time average over all fields in a file.

Parameters:

  • histfile (str) – file to perform time average on

  • newf (str) – path to target file

  • avgtime (bool) – whether histfile has averaged fields (e.g., atmos_average)

  • avgtod (bool) – whether histfile has a diurnal time dimenion (e.g., atmos_diurn)

  • bin_period (int, optional) – the time binning period if histfile has averaged fields (i.e., if avgtime==True), defaults to 5

Returns:

a time-averaged file

Return type:

None

Raises:

  • KeyError – if the required variables are not found

  • ValueError – if the required dimensions are not found

  • AttributeError – if the required attributes are not found

Note::

The diurn file is created by binning the Legacy files. The average and daily files are created by averaging over the diurn file.

ls2sol_1year(Ls_deg, offset=True, round10=True)

Returns a sol number from the solar longitude.

This is consistent with the MGCM model. The Ls is the solar longitude in degrees. The sol number is the number of sols since the perihelion (Ls = 250.99 degrees).

Parameters:

  • Ls_deg (float) – solar longitude [°]

  • offset (bool) – if True, force year to start at Ls 0

  • round10 (bool) – if True, round to the nearest 10 sols

Returns:

Ds the sol number

Return type:

float

Raises:

  • ValueError – if the required variables are not found

  • KeyError – if the required variables are not found

  • AttributeError – if the required attributes are not found

..note::

This is consistent with 0 <= Ls <= 359.99, but not for monotically increasing Ls.

main()

Main entry point for MarsFiles data processing utility.

This function processes input NetCDF or legacy MGCM files according to command-line arguments. It supports a variety of operations, including:

  • Conversion of legacy MGCM files to FV3 format.

  • Concatenation and splitting of NetCDF files along specified dimensions.

  • Temporal binning of daily files into average or diurnal files.

  • Temporal filtering (high-pass, low-pass, band-pass) using spectral methods.

  • Spatial (zonal) filtering and decomposition using spherical harmonics.

  • Tidal analysis and harmonic decomposition of diurnal files.

  • Regridding of data to match a target NetCDF file’s grid.

  • Zonal averaging of variables over longitude.

The function handles file path resolution, argument validation, and orchestrates the appropriate processing routines based on user-specified options. Output files are written in NetCDF format, with new variables and dimensions created as needed.

Global Variables:

data_dir (str): The working directory for input/output files.

Arguments:

None directly. Uses global ‘args’ parsed from command-line.

Returns:

None. Outputs are written to disk.

Raises:

SystemExit: For invalid argument combinations or processing errors.

make_FV3_files(fpath, typelistfv3, renameFV3=True)

Make MGCM-like average, daily, and diurn files.

Used if call to [-bin --bin_files] is made AND Legacy files are in netCDF format (not fort.11).

Parameters:

  • fpath (str) – Full path to the Legacy netcdf files

  • typelistfv3 (list) – MGCM-like file type: average, daily, or diurn

  • renameFV3 (bool) – Rename the files from Legacy_LsXXX_LsYYY.nc to XXXXX.atmos_average.nc following MGCM output conventions

Returns:

The MGCM-like files: XXXXX.atmos_average.nc, XXXXX.atmos_daily.nc, XXXXX.atmos_diurn.nc.

Return type:

None

Note::

The average and daily files are created by averaging over the diurn file. The diurn file is created by binning the Legacy files.

Note::

The diurn file is created by binning the Legacy files.

process_time_shift(file_list)

Converts diurn files to local time.

This function is used to convert diurn files to local time.

Parameters:

file_list (list) – list of file names

Returns:

None

Return type:

None

Raises:

  • OSError – If the file cannot be removed.

  • IOError – If the file cannot be moved.

  • Exception – If the file cannot be opened.

  • ValueError – If the file cannot be accessed.

  • TypeError – If the file is not of the correct type.

  • IndexError – If the file does not have the correct index.

replace_at_index(tuple_dims, idx, new_name)

Replaces the dimension at the given index with a new name.

If new_name is None, the dimension is removed. This is designed to work specifically with LegacyCGM.nc files.

Parameters:

  • tuple_dims (tuple) – the dimensions as tuples e.g. (pfull, nlat, nlon)

  • idx (int) – index indicating axis with the dimensions to update (e.g. idx = 1 for nlat)

  • new_name (str) – new dimension name (e.g. latitude)

Returns:

updated dimensions

Return type:

tuple

Raises:

  • KeyError – if the required variables are not found

  • ValueError – if the required dimensions are not found

  • AttributeError – if the required attributes are not found

replace_dims(dims, todflag)

Replaces dimensions with MGCM-like names. Removes time_of_day. This is designed to work specifically with LegacyCGM.nc files.

Parameters:

  • dims (str) – dimensions of the variable

  • todflag (bool) – indicates whether there exists a time_of_day dimension

Returns:

new dimension names for the variable

Return type:

tuple

Raises:

  • KeyError – if the required variables are not found

  • ValueError – if the required dimensions are not found

  • AttributeError – if the required attributes are not found

split_files(file_list, split_dim)

Extracts variables in the file along the specified dimension.

The function creates a new file with the same name as the input file, but with the specified dimension sliced to the requested value or range of values. The new file is saved in the same directory as the input file.

The function also checks the bounds of the requested dimension against the actual values in the file. If the requested value is outside the bounds of the dimension, an error message is printed and the function exits.

The function also checks if the requested dimension is valid (i.e., time, lev, lat, or lon). If the requested dimension is invalid, an error message is printed and the function exits.

The function also checks if the requested dimension is a single dimension (i.e., areo). If the requested dimension is a single dimension, the function removes all single dimensions from the areo dimension (i.e., scalar_axis) before performing the extraction.

Parameters:

file_list – list of file names

Returns:

new file with sliced dimensions

Return type:

None

Raises:

  • OSError – If the file cannot be removed.

  • IOError – If the file cannot be moved.

  • Exception – If the file cannot be opened.

  • ValueError – If the file cannot be accessed.

  • TypeError – If the file is not of the correct type.

  • IndexError – If the file does not have the correct index.

all_args
args
debug
exit_code
out_ext
out_ext
parser