alex.utils package

Submodules

alex.utils.analytics module

alex.utils.audio module

alex.utils.audio_play module

alex.utils.autopath module

self cloning, automatic path configuration

copy this into any subdirectory of pypy from which scripts need to be run, typically all of the test subdirs. The idea is that any such script simply issues

import autopath

and this will make sure that the parent directory containing “pypy” is in sys.path.

If you modify the master “autopath.py” version (in pypy/tool/autopath.py) you can directly run it which will copy itself on all autopath.py files it finds under the pypy root directory.

This module always provides these attributes:

pypydir pypy root directory path this_dir directory where this autopath.py resides

alex.utils.cache module

class alex.utils.cache.Counter[source]

Bases: dict

Mapping where default values are zero

alex.utils.cache.get_persitent_cache_content(key)[source]
alex.utils.cache.lfu_cache(maxsize=100)[source]

Least-frequently-used cache decorator.

Arguments to the cached function must be hashable. Cache performance statistics stored in f.hits and f.misses. Clear the cache with f.clear(). http://en.wikipedia.org/wiki/Least_Frequently_Used

alex.utils.cache.lru_cache(maxsize=100)[source]

Least-recently-used cache decorator.

Arguments to the cached function must be hashable. Cache performance statistics stored in f.hits and f.misses. Clear the cache with f.clear(). http://en.wikipedia.org/wiki/Cache_algorithms#Least_Recently_Used

alex.utils.cache.persistent_cache(method=False, file_prefix='', file_suffix='')[source]

Persistent cache decorator.

It grows indefinitely. Arguments to the cached function must be hashable. Cache performance statistics stored in f.hits and f.misses.

alex.utils.cache.set_persitent_cache_content(key, value)[source]

alex.utils.caminfodb module

class alex.utils.caminfodb.CamInfoDb(db_path)[source]

Bases: object

get_by_id(rec_id)[source]
get_matching(query)[source]
get_possible_values()[source]
get_slots()[source]
matches(rec, query)[source]

alex.utils.config module

class alex.utils.config.Config(file_name=None, project_root=False, config=None)[source]

Bases: object

Config handles configuration data necessary for all the components in Alex. It is implemented using a dictionary so that any component can use arbitrarily structured configuration data.

Before the configuration file is loaded, it is transformed as follows:

  1. ‘{cfg_abs_path}’ as a string anywhere in the file is replaced by an

    absolute path of the configuration files. This can be used to make the configuration file independent of the location of programs that use it.

DEFAULT_CFG_PPATH = u'resources/default.cfg'
config_replace(p, s, d=None)[source]

Replace a pattern p with string s in the whole config (recursively) or in a part of the config given in d.

contains(*path)[source]

Check if configuration contains given keys (= path in config tree).

get(i, default=None)[source]
getpath(path, default=None)[source]
load(file_name)[source]
classmethod load_configs(config_flist=[], use_default=True, log=True, *init_args, **init_kwargs)[source]

Loads and merges configs from paths listed in `config_flist’. Use this method instead of direct loading configs, as it takes care of not only merging them but also processing some options in a special way.

Arguments:
config_flist – list of paths to config files to load and merge;
order matters (default: [])
use_default – whether to insert the default config
($ALEX/resources/default.cfg) at the beginning of `config_flist’ (default: True)
log – whether to log the resulting config using the system logger
(default: True)
init_args – additional positional arguments will be passed to
constructors for each config
init_kwargs – additional keyword arguments will be passed to
constructors for each config
load_includes()[source]
merge(other)[source]

Merges self’s config with other’s config and saves it as a new self’s config.

Keyword arguments:
  • other: a Config object whose configuration dictionary to merge

    into self’s one

unfold_lists(pattern, unfold_id_key=None, part=[])[source]

Unfold lists under keys matching the given pattern into several config objects, each containing one item. If pattern is None, all lists are expanded.

Stores a string representation of the individual unfolded values under the unfold_id_key if this parameter is set.

Only expands a part of the whole config hash (given by list of keys forming a path to this part) if the path parameter is set.

update(new_config, config_dict=None)[source]

Updates the nested configuration dictionary by another, potentially also nested dictionary.

Keyword arguments:
  • new_config: the new dictionary to update with
  • config_dict: the config dictionary to be updated
alex.utils.config.as_project_path(path)[source]
alex.utils.config.callback_download_progress(blocks, block_size, total_size)[source]

callback function for urlretrieve that is called when connection is created and when once for each block

Parameters:
  • blocks – number of blocks transferred so far
  • block_size – in bytes
  • total_size – in bytes, can be -1 if server doesn’t return it
alex.utils.config.is_update_server_reachble()[source]
alex.utils.config.load_as_module(path, force=False, encoding=u'UTF-8', text_transforms=[])[source]

Loads a file pointed to by `path’ as a Python module with minimal impact on the global program environment. The file name should end in ‘.py’.

Arguments:

path – path towards the file force – whether to load the file even if its name does not end in

‘.py’

encoding – character encoding of the file text_transforms – collection of functions to be run on the original

file text

Returns the loaded module object.

alex.utils.config.online_update(file_name)[source]

This function can download file from a default server if it is not available locally. The default server location can be changed in the config file.

The original file name is transformed into absolute name using as_project_path function.

Parameters:file_name – the file name which should be downloaded from the server
Returns:a file name of the local copy of the file downloaded from the server
alex.utils.config.set_online_update_server(server_name)[source]

Set the name of the online update server. This function can be used to change the server name from inside a config file.

Parameters:server_name – the HTTP(s) path to the server and a location where the desired data reside.
Returns:None
alex.utils.config.to_project_path(path)[source]

Converts a relative or absoulute file system path to a path relative to project root.

alex.utils.cuda module

alex.utils.cuda.cudasolve(A, b, tol=0.001, normal=False, regA=1.0, regI=0.0)[source]

Conjugate gradient solver for dense system of linear equations.

Ax = b

Returns: x = A^(-1)b

If the system is normal, then it solves

(regA*A’A +regI*I)x= b

Returns: x = (A’A +reg*I)^(-1)b

alex.utils.czech_stemmer module

Czech stemmer Copyright © 2010 Luís Gomes <luismsgomes@gmail.com>.

Ported from the Java implementation available at:
http://members.unine.ch/jacques.savoy/clef/index.html
alex.utils.czech_stemmer.cz_stem(l, aggressive=False)[source]
alex.utils.czech_stemmer.cz_stem_word(word, aggressive=False)[source]

alex.utils.enums module

alex.utils.enums.enum(*sequential, **named)[source]

Useful for creating enumerations.

e.g.: DialogueType = enum(deterministic=0, statistical=1, mix=2)

alex.utils.env module

alex.utils.env.root()[source]

Finds the root of the project and return it as string.

The root is the directory named alex.

alex.utils.excepthook module

Depending on the hook_type, ExceptionHook class adds various hooks how to catch exceptions.

class alex.utils.excepthook.ExceptionHook(hook_type, logger=None)[source]

Bases: object

Singleton objects for registering various hooks for sys.exepthook. For registering a hook, use set_hook.

apply()[source]

The object can be used to store settings for excepthook. a = ExceptionHook(‘log’) # now it logs b = ExceptionHook(‘ipdb’) # now it uses ipdb a.apply() # now it logs again

logger = None
classmethod set_hook(hook_type=None, logger=None)[source]

Choose an exception hook from predefined functions.

hook_type: specify the name of the hook method

alex.utils.excepthook.hook_decorator(f)[source]

Print the caution message when the decorated function raises an error.

alex.utils.excepthook.ipdb_hook(*args, **kwargs)[source]
alex.utils.excepthook.log_and_ipdb_hook(*args, **kwargs)[source]
alex.utils.excepthook.log_hook(*args, **kwargs)[source]

alex.utils.exceptions module

exception alex.utils.exceptions.ConfigException[source]

Bases: alex.AlexException

exception alex.utils.exceptions.SessionClosedException[source]

Bases: alex.AlexException

exception alex.utils.exceptions.SessionLoggerException[source]

Bases: alex.AlexException

alex.utils.exdec module

alex.utils.exdec.catch_ioerror(user_function, msg='')[source]

alex.utils.filelock module

Context manager for locking on a file. Obtained from

http://www.evanfosmark.com/2009/01 /cross-platform-file-locking-support-in-python/,

licensed under BSD.

This is thought to work safely on NFS too, in contrast to fcntl.flock(). This is also thought to work safely over SMB and else, in contrast to fcntl.lockf(). For both issues, consult http://oilq.org/fr/node/13344.

Use as simply as

with FileLock(filename):
<critical section for working with the file at `filename’>
class alex.utils.filelock.FileLock(file_name, timeout=10, delay=0.05)[source]

Bases: object

A file locking mechanism that has context-manager support so you can use it in a with statement. This should be relatively portable as it doesn’t rely on msvcrt or fcntl for the locking.

acquire()[source]

Acquire the lock, if possible. If the lock is in use, it check again every `wait’ seconds. It does this until it either gets the lock or exceeds `timeout’ number of seconds, in which case it throws an exception.

release()[source]

Get rid of the lock by deleting the lockfile. When working in a `with’ statement, this method gets automatically called at the end.

exception alex.utils.filelock.FileLockException[source]

Bases: exceptions.Exception

alex.utils.fs module

Filesystem utility functions.

class alex.utils.fs.GrepFilter(stdin, stdout, breakchar=u'n')[source]

Bases: multiprocessing.process.Process

add_listener(regex, callback)[source]

Adds a listener to the output strings.

Arguments:
regex – the compiled regular expression to look for
(`regex.search’) in any piece of output
callback – a callable that is invoked for output where `regex’ was

found. This will be called like this:

outputting &= callback(output_unicode_str)

That means, callback should take the unicode string argument containing what would have been output and return a boolean value which is True iff outputting should stop.

Returns the index of the listener for later reference.

flush(force=True)[source]
remove_listener(listener_idx)[source]
run()[source]
write(unistr)[source]
alex.utils.fs.find(dir_, glob_, mindepth=2, maxdepth=6, ignore_globs=[], ignore_paths=None, follow_symlinks=True, prune=False, rx=None, notrx=None)[source]

A simplified version of the GNU `find’ utility. Lists files with basename matching `glob_‘ found in `dir_‘ in depth between `mindepth’ and `maxdepth’.

The `ignore_globs’ argument specifies a glob for basenames of files to be ignored. The `ignore_paths’ argument specifies a collection of real absolute pathnames that are pruned from the search. For efficiency reasons, it should be a set.

In the current implementation, the traversal resolves symlinks before the file name is checked. However, taking symlinks into account can be forbidden altogether by specifying `follow_symlinks=False’. Cycles during the traversal are avoided.

  • prune: whether to prune the subtree below a matching directory

  • rx: regexp to use as an additional matching criterion apart from

    `glob_‘; the `re.match’ function is used, as opposed to `re.find’

  • notrx: like `rx’ but this specifies the regexp that must NOT match

The returned set of files consists of real absolute pathnames of those files.

alex.utils.fs.normalise_path(path)[source]

Normalises a filesystem path using tilde expansion, absolutising and normalising the path, and resolving symlinks.

alex.utils.fs.test_grep_filter()[source]

alex.utils.htk module

alex.utils.interface module

class alex.utils.interface.Interface[source]

Bases: object

alex.utils.interface.interface_method(f)[source]

alex.utils.lattice module

alex.utils.mfcc module

alex.utils.mproc module

Implements useful classes for handling multiprocessing implementation of the Alex system.

class alex.utils.mproc.InstanceID[source]

Bases: object

This class provides unique ids to all instances of objects inheriting from this class.

get_instance_id(*args, **kw)[source]
instance_id = <Synchronized wrapper for c_int(0)>
lock = <Lock(owner=None)>
class alex.utils.mproc.SystemLogger(output_dir, stdout_log_level='DEBUG', stdout=True, file_log_level='DEBUG')[source]

Bases: object

This is a multiprocessing-safe logger. It should be used by all components in Alex.

critical(*args, **kwargs)[source]
debug(*args, **kwargs)[source]
error(*args, **kwargs)[source]
exception(message)[source]
formatter(*args, **kw)[source]

Format the message - pretty print

get_session_dir_name(*args, **kw)[source]

Return directory where all the call related files should be stored.

get_time_str()[source]

Return current time in dashed ISO-like format.

It is useful in constructing file and directory names.

info(*args, **kwargs)[source]
levels = {'INFO': 20, 'CRITICAL': 40, 'EXCEPTION': 50, 'SYSTEM-LOG': 0, 'WARNING': 30, 'ERROR': 60, 'DEBUG': 10}
lock = <RLock(None, 0)>
log(*args, **kw)[source]

Logs the message based on its level and the logging setting. Before writing into a logging file, it locks the file.

session_end(*args, **kw)[source]

WARNING: Deprecated Disables logging into the session-specific directory.

We better do not end a session because very often after the session_end() method is called there are still incoming messages. Therefore, it is better to wait for the session_start() method to set a new destination for the session log.

session_start(*args, **kw)[source]

Create a specific directory for logging a specific call.

NOTE: This is not completely safe. It can be called from several processes.

session_system_log(*args, **kwargs)[source]

This logs specifically only into the call-specific system log.

warning(*args, **kwargs)[source]
alex.utils.mproc.async(func)[source]

A function decorator intended to make “func” run in a separate thread (asynchronously). Returns the created Thread object

E.g.: @async def task1():

do_something

@async def task2():

do_something_too

t1 = task1() t2 = task2() ... t1.join() t2.join()

alex.utils.mproc.etime(name='Time', min_t=0.3)[source]

This decorator measures the execution time of the decorated function.

alex.utils.mproc.file_lock(file_name)[source]

Multiprocessing lock using files. Lock on a specific file.

alex.utils.mproc.file_unlock(lock_file)[source]

Multiprocessing lock using files. Unlock on a specific file.

alex.utils.mproc.global_lock(lock)[source]

This decorator makes the decorated function thread safe.

Keyword arguments:
lock – a global variable pointing to the object to lock on
alex.utils.mproc.local_lock()[source]

This decorator makes the decorated function thread safe.

For each function it creates a unique lock.

alex.utils.nose_plugins module

alex.utils.parsers module

class alex.utils.parsers.CamTxtParser(lower=False)[source]

Bases: object

Parser of files of the following format: <<BOF>> [record]

[record]

... <<EOF>>

where [record] has the following format:

<<[record]>> [property name]([property value]) <</[record]>>

[property name] and [property value] are arbitrary strings

Any ” or ‘ characters are stripped from the beginning and end of each [property value].

line_expr = <_sre.SRE_Pattern object>
parse(f_obj)[source]

Parse the given file and return list of dictionaries with parsed values.

Arguments: f_obj – filename of file or file object to be parsed

alex.utils.procname module

alex.utils.procname.get_proc_name()[source]
alex.utils.procname.set_proc_name(newname)[source]

alex.utils.rdb module

class alex.utils.rdb.Rdb(port=4446)[source]

Bases: pdb.Pdb

do_c(arg)
do_cont(arg)
do_continue(arg)[source]

alex.utils.sessionlogger module

class alex.utils.sessionlogger.SessionLogger[source]

Bases: multiprocessing.process.Process

This is a multiprocessing-safe logger. It should be used by Alex to log information according the SDC 2010 XML format.

Date and times should also include time zone.

Times should be in seconds from the beginning of the dialogue.

cancel_join_thread()[source]
run()[source]
set_cfg(cfg)[source]
set_close_event(close_event)[source]

alex.utils.test_analytics module

alex.utils.test_fs module

Unit tests for alex.util.fs.

class alex.utils.test_fs.TestFind(methodName='runTest')[source]

Bases: unittest.case.TestCase

setUp()[source]

Creates a playground of a directory tree. It looks like this: <testroot>/

  • a/
    • aa/

    • ab/

    • ac/
      • aca/
        • acaa/
        • acab -> baaaa
  • b/
    • ba/
      • baa/
        • baaa/
          • baaaa -> daaa

          • baaab/
            • baaaba/
              • baaabaa/
              • baaabab -> ca
  • c/
    • ca/
      • caa/

      • cab/
        • caba/
      • cac -> db

  • d/
    • da/
      • daa/
        • daaa -> acab
    • db -> baaaba

tearDown()[source]

Deletes the mock-up directory tree.

test_cycles()[source]

Test the processing of cycles in the directory structure.

test_depth()[source]

Tests mindepth and maxdepth.

test_globs()[source]

Tests processing of the selection glob.

test_ignore_globs()[source]

Test the functionality of ignore globs.

test_symlinks1()[source]

Basic test for symlinks.

test_wrong_args()[source]

Test for handling wrong arguments.

alex.utils.test_sessionlogger module

alex.utils.test_text module

class alex.utils.test_text.TestString(methodName='runTest')[source]

Bases: unittest.case.TestCase

test_parse_command()[source]
test_split_by()[source]

alex.utils.text module

class alex.utils.text.Escaper(chars=u''"', escaper=u'\', re_flags=0)[source]

Bases: object

Creates a customised escaper for strings. The characters that need escaping, as well as the one used for escaping can be specified.

ESCAPED = 1
ESCAPER = 0
NORMAL = 2
annotate(esced)[source]

Annotates each character of a text that has been escaped whether:

Escaper.ESCAPER - it is the escape character Escaper.ESCAPED - it is a character that was escaped Escaper.NORMAL - otherwise.

It is expected that only parts of the text may have actually been escaped.

Returns a list with the annotation values, co-indexed with characters of the input text.

escape(text)[source]

Escapes the text using the parameters defined in the constructor.

static re_literal(char)[source]

Escapes the character so that when it is used in a regexp, it matches itself.

static re_literal_list(chars)[source]

Builds a [] group for a regular expression that matches exactly the characters specified.

unescape(text)[source]

Unescapes the text using the parameters defined in the constructor.

alex.utils.text.escape_special_characters_shell(text, characters=u'\'"')[source]

Simple function that tries to escape quotes. Not guaranteed to produce the correct result!! If that is needed, use the new `Escaper’ class.

alex.utils.text.findall(text, char, start=0, end=-1)[source]
alex.utils.text.min_edit_dist(target, source)[source]

Computes the min edit distance from target to source.

alex.utils.text.min_edit_ops(target, source, cost=<function <lambda>>)[source]

Computes the min edit operations from target to source.

Parameters:
  • target – a target sequence
  • source – a source sequence
  • cost – an expression for computing cost of the edit operations
Returns:

a tuple of (insertions, deletions, substitutions)

alex.utils.text.parse_command(command)[source]

Parse the command name(var1=”val1”,...) into a dictionary structure:

E.g. call(destination=”1245”,opt=”X”) will be parsed into:

{ “__name__”: “call”,
“destination”: “1245”, “opt”: “X”}

Return the parsed command in a dictionary.

alex.utils.text.split_by(text, splitter, opening_parentheses=u'', closing_parentheses=u'', quotes=u'\'"')[source]

Splits the input text at each occurrence of the splitter only if it is not enclosed in parentheses.

text - the input text string splitter - multi-character string which is used to determine the position

of splitting of the text
opening_parentheses - an iterable of opening parentheses that has to be
respected when splitting, e.g. “{(” (default: ‘’)
closing_parentheses - an iterable of closing parentheses that has to be
respected when splitting, e.g. “})” (default: ‘’)

quotes - an iterable of quotes that have to come in pairs, e.g. ‘”’

alex.utils.text.split_by_comma(text)[source]

alex.utils.token module

alex.utils.token.get_token(cfg)[source]

alex.utils.ui module

alex.utils.ui.getTerminalSize()[source]

Retrieves the size of the current terminal window.

Returns (None, None) in case of lack of success.

alex.utils.various module

alex.utils.various.crop_to_finite(val)[source]
alex.utils.various.flatten(list_, ltypes=(<type 'list'>, <type 'tuple'>))[source]

Flatten nested list into a simple list.

alex.utils.various.get_text_from_xml_node(node)[source]

Get text from all child nodes and concatenate it.

alex.utils.various.group_by(objects, attrs)[source]

Groups `objects’ by the values of their attributes `attrs’.

Returns a dictionary mapping from a tuple of attribute values to a list of objects with those attribute values.

class alex.utils.various.nesteddict[source]

Bases: collections.defaultdict

walk()[source]
alex.utils.various.remove_dups_stable(l)[source]

Remove duplicates from a list but keep the ordering.

@return: Iterator over unique values in the list

alex.utils.various.split_to_bins(A, S=4)[source]

Split the A array into bins of size N.

Module contents

class alex.utils.DummyLogger[source]

Bases: object

alex.utils.one()[source]
alex.utils.script_path(fname, *args)[source]

Return path relative to the directory of the given file, and join the additional path parts.

Args:
fname (str): file used to determine the root directory args (list): additional path parts