command_interpreter – The command line interpreter

The core functionality

This module implements the core of the command interpreter and any part essential for running it

class vdat.command_interpreter.core.CommandInterpreter(command, command_config, selected=None)

Interpret and execute the command.

See The interpreter section in the documentation for more details

All the custom errors are defined in vdat.command_interpreter.exceptions. The ones raised in the constructor are derived from from CIValidationError,

Parameters:

command : string

command to parse

command_config : dict

dictionary containing the instructions to execute the command. A deep copy is executed

selected : list-like, optional

None or a list of the selected items; if None no filtering of the primary files is done; otherwise must be an object supporting the membership test operator in.

Raises:

CINoExeError

if the executable does not exists

CIParseError

if there is some error when extracting the keywords

CIKeywordError

for missing keywords or for keywords of wrong type

CIKeywordTypeError

if the type of the keywords is not among the known ones

primary_types = <vdat.command_interpreter.types.PrimaryTypes object at 0x7eff51f034d0>

keyword_types = <vdat.command_interpreter.types.KeywordTypes object at 0x7eff51f03510>

execute_types = <vdat.command_interpreter.types.ExecuteTypes object at 0x7eff51f03550>

run()

Collect the files, expand and run the required command

All the custom errors raised here derive from CIRunError.

Raises:

CICommandFmtError

if the substitution of the command doesn’t work

_replace_alias()

If the command configuration has the is_alias_of replace the executable name

_check_exe()

Check that the executable can be found and replace it with the full path returned by distutils.spawn.find_executable()

_get_keys(command)

Extract the keywords from the command

_validate_mandatory()

Check that all the mandatory keywords are provided.

If mandatory is not found, return without doing anything

_validate_keywords()

Check that all the requested keywords are in the configuration

_key_types(keys)

Scan the keys and check that the interpreter knows how to deal with them.

Parameters:

keys : list of strings

keys extracted from the command

Returns:

primary_func : callable

function to call to get the list of primary files

keyword_map : dictionary

map non-primary keywords to the function used to handle them

Raises:

CIKeywordTypeError

if the type of the keyword is not known

_get_value_as_dict(key)

Get the value of key from the configuration. If it’s a string, convert it to a dictionary with two entries:

• type: plain
• value: value

and re-add it in the configuration dictionary

Parameters:

key : string

key to get

Returns:

value : dictionary

dictionary defining the type

Raises:

CIKeywordError

if value is not a dictionary or a string

_filter_selected()

Look for the existence of the filter_selected option and check that it is of the correct type and that selected is of the correct type

Returns:

filter_func : function

function that accepts one string (one element of the primary list) and returns True or False if that element is valid or not

_execute()

Look for the existence of the execute option in the configuration and check that it is of the correct type.

Returns:

execute_func : function

function that accepts one string (one element of the primary list) and returns True or False if that element must be run or not

_run(index, primary)

Job that create the keywords, create and run the command string and log it

Parameters:

index : int

index of the current job

primary : string

primary file(s) to pass to the functions handling the keywords

_true(*_, **__)

returns always true

Helpers

Helper function and classes.

These functionalities are not essential for the interpreter, but can help the to setup things

Types

Define enumerate-like classes that allows to map from keys to key types and to the functions that needs to be called to deal with any of them.

It uses pkg_resources and entry points to make the framework extendible

vdat.command_interpreter.types._load_entrypoints(group)

Get all the entry points for the group and load them.

Parameters:

group : string

name of the group to load

Returns:

entry_points : dictionary

key: name; value: callable loaded from the entry point

class vdat.command_interpreter.types._Types

Base class for the types. It shouldn’t be used directly.

If a type loop exists, it can be accessed as instance.loop or instance['loop']

Attributes

known_types()

return the list of known types

entry_point_group = None

known_types()

return the list of known types

class vdat.command_interpreter.types.PrimaryTypes

Fill the type<–>function mapping using the vdat.cit.primary entry point.

entry_point_group = 'vdat.cit.primary'

class vdat.command_interpreter.types.KeywordTypes

Fill the type<–>function mapping using the vdat.cit.keyword entry point.

entry_point_group = 'vdat.cit.keyword'

class vdat.command_interpreter.types.ExecuteTypes

Fill the type<–>function mapping using the vdat.cit.execute entry point.

entry_point_group = 'vdat.cit.execute'

vdat.command_interpreter.types.primary_template(target_dir, key_val)

Template for a function that deals with a primary keyword.

It collects the files from the target_dir according to the instructions in key_val, if any and either yield a value or return an iterable.

Parameters:

target_dir : string

directory in which the files must be collected

key_val : dictionary

configuration for the key handle

Returns:

yield a string or iterable of strings

Raises:

CIPrimaryError

if something goes wrong when handling the primary key

vdat.command_interpreter.types.keyword_template(primary, key_val)

Template for a function that deals with a non-primary keyword.

A keyword has a value either statically stored in key_val or its value need to be extracted from the value of the primary file(s).

Parameters:

primary : string

the value of one of the items returned by primary_template()

key_val : dictionary

configuration for the key handle

Returns:

string

value to associate to the keyword

Raises:

CIKeywordError

if something goes wrong when handling the key

vdat.command_interpreter.types.execute_template(primary, config)

For each of the primary entry, this function is called to decide whether to execute or skip the command.

Parameters:

primary : string

the value of one of the items returned by primary_template()

config : dictionary

configuration for the command

Returns:

bool

True: the command is executed; False: the command is skipped

vdat.command_interpreter.types.primary_plain(target_dir, key_val)

Get all the files in target_dir matching the string in key_val['value']

Parameters:

target_dir : string

directory in which the files must be collected

key_val : dictionary

configuration for the key handle

Returns:

iterator

yields file names matching the value recursively

vdat.command_interpreter.types.primary_loop(target_dir, key_val)

Make a nested loop over the set of given keys, in each step of the loop construct the value using python format string syntax and then get all the files matching it.

If any of the steps doesn’t produce any file, no value is yielded.

Parameters:

target_dir : string

directory in which the files must be collected

key_val : dictionary

configuration for the key handle

Returns:

yields a string of space separated file names

vdat.command_interpreter.types.primary_groupby(target_dir, key_val)

Loop over all the files matching the value entry. For each one, create a list of file names replacing the regex in pattern with the elements of replace.

Parameters:

target_dir : string

directory in which the files must be collected

key_val : dictionary

configuration for the key handle

Returns:

yields a string of space separated file names

vdat.command_interpreter.types.keyword_plain(_, key_val)

Returns the value contained in the keyword

Parameters:

primary : string

ignored

key_val : dictionary

configuration for the key handle

Returns:

string

value to associate to the keyword

vdat.command_interpreter.types.keyword_header(primary, key_val)

Extract and parse an fits header keyword from the first file.

Extract the value keyword from the header. If extract is in the configuration, it instruct hos to build a variable out of the extracted header value.

Parameters:

primary : string

primary file name(s)

key_val : dictionary

configuration for the key handle

Returns:

string

value to associate to the keyword

vdat.command_interpreter.types.keyword_format(primary, key_val)

Create a new string formatting value according to the provided keys.

The keys are substituted using format string syntax.

The value of keys is a map between values to substitute in value and keyword types used to extract them from the primary file name. Strings are interpreted as of type plain.

Parameters:

primary : string

primary file name(s)

key_val : dictionary

configuration for the key handle

Returns:

string

value to associate to the keyword

vdat.command_interpreter.types.keyword_regex(primary, key_val)

Extract a string from the primary using regular expression substitution

Parameters:

primary : string

primary file name(s)

key_val : dictionary

configuration for the key handle

Returns:

string

string built from the primary file name

vdat.command_interpreter.types.execute_new_file(primary, config)

For each of the primary entry, it constructs a string using the keyword type defined by subtype. If that string corresponds to something existing in the file system, returns False.

Parameters:

primary : string

the value of one of the items returned by primary_template()

config : dictionary

configuration for the command (not for the type)

Returns:

bool

True: if the output of the keyword handling does not exist

vdat.command_interpreter.types._to_number(string)

Convert the string to a number. It first tries to covert it to an int and then to a float. If it fails gives up.

Parameters:

string : string

string to convert

Returns:

int or float

converted value

Raises:

ValueError

if the conversion fails

Exceptions

Command interpreter exceptions

exception vdat.command_interpreter.exceptions.CIError

Bases: exceptions.Exception

Generic exception. It’s the parent of all the other exceptions defined here

exception vdat.command_interpreter.exceptions.CIValidationError

Bases: vdat.command_interpreter.exceptions.CIError

Exception raised when validating the command in the constructor

exception vdat.command_interpreter.exceptions.CINoExeError(name)

Bases: vdat.command_interpreter.exceptions.CIValidationError

Raised when the executable name is not found

exception vdat.command_interpreter.exceptions.CIParseError

Bases: vdat.command_interpreter.exceptions.CIValidationError

Failed parsing of the command

exception vdat.command_interpreter.exceptions.CIKeywordValidationError

Bases: vdat.command_interpreter.exceptions.CIValidationError

Raised when the keyword validation fails

exception vdat.command_interpreter.exceptions.CIKeywordTypeError

Bases: vdat.command_interpreter.exceptions.CIKeywordValidationError

Raised when the keyword doesn’t have a type key or it’s type is not known

exception vdat.command_interpreter.exceptions.CIRunError

Bases: vdat.command_interpreter.exceptions.CIError

Raised when running the command

exception vdat.command_interpreter.exceptions.CIPrimaryError

Bases: vdat.command_interpreter.exceptions.CIRunError

Raised if something bad happens when handling a primary keyword

exception vdat.command_interpreter.exceptions.CIKeywordError

Bases: vdat.command_interpreter.exceptions.CIRunError

Raised if something bad happens when handling a secondary keyword

exception vdat.command_interpreter.exceptions.CICommandFmtError

Bases: vdat.command_interpreter.exceptions.CIRunError

Raised when the replacement of the keyword fails

Relays

Except for the logging mechanism, the CommandInterpreter uses relays-like objects to communicate with the external word. This module defines a few classes with an emit method, that mimic PyQt signals

The names of the emit method arguments are the type of the parameter followed by an underscore and optionally by an explanatory name.

Available relays:

• command_string: accept an int and a string

  _CommandString.emit(int_, string_)

  Default implementation: print the string_ for when int_ == 0

• progress: accept three numbers, the total expected number, the number of successes and of failures;

  _Progress.emit(int_tot, int_done, int_fail)

  Default implementation: Print the percentages of finished, successful and failed jobs, overwriting the line

• logger: accept a two strings

  _Logger.emit(int_level, string_msg)

  Default implementation: print int_level: string_msg

  The int_level values are chosen to be the standard logging levels and are converted to strings accordingly

  The idea behind this relay is to bind it to the main logger if needed. We decided not to do this directly as we don’t know the name of the logger. We implement it in this way to have a more coherent interface.

vdat.command_interpreter.relay.override_emit(name, new_emit)

Replace the emit method in the relay instance called name with new_emit. The original class is not modified.

Parameters:

name : string

name of the relay

new_emit : callable

function to use to replace the default emit method. This function must have at least one argument, self and it’s signature must match the original one to avoid errors at runtime