Welcome to Alex Dialogue Systems Framework’s documentation!¶

The data¶

To build the domain specific language model, we use the approach described in Approach to bootstraping the domain specific language models. So far, we have collected this data:

1. selected out-of-domain data - more than 2000 sentences
2. bootstrap text - 289 sentences
3. indomain data - more than 9000 sentences (out of which about 900 of the sentences are used as development data)

Building the models¶

The models are built using the build.py script.

It requires to set the following variables:

bootstrap_text                  = "bootstrap.txt"
classes                         = "../data/database_SRILM_classes.txt"
indomain_data_dir               = "indomain_data"

The variables description:

• bootstrap_text - the bootstrap.txt file contains handcrafted in-domain sentences.
• classes - the ../data/database_SRILM_classes.txt file is created by the database.py script in the alex/applications/PublicTransportInfoCS/data directory.
• indomain_data_dir - should include links to directories containing asr_transcribed.xml files with transcribed audio data

The process of building/re-building the LM is:

cd ../data
./database.py dump
cd ../lm
./build.py

Distributions of the models¶

The final.* models are large. Therefore, they should be distributed online on-demand using the online_update function. Please do not forget to place the models generated by the ./build.py script on the distribution servers.

Reuse of build.py¶

The build.py script can be easily generalised to a different language or different text data, e.g. the in-domain data.

Running the system at UFAL with the full UFAL access¶

There are multiple configuration that can used to run the system. In general, it depends on what components you want go use and on what telephone extension you want to run the system.

Within UFAL, we run the system using the following commands:

• vhub_live - deployment of our live system on our toll-free phone number, with the default configuration
• vhub_live_b1 - a system deployed to backup the system above
• vhub_live_b2 - a system deployed to backup the system above
• vhub_live_kaldi - a version of our live system explicitly using Kaldi ASR

To test the system we use:

• vhub_test - default test version of our system deployed on our test extension, logging locally into ../call_logs
• vhub_test_google_only - test version of our system on our test extension, using Google ASR, TTS, Directions, logging locally into ../call_logs
• vhub_test_google_kaldi - test version of our system on our test extension, using Google TTS, Directions, and Kaldi ASR, logging locally into ../call_logs
• vhub_test_hdc_slu - default test version of our system deployed on our test extension, using HDC SLU, logging locally into ../call_logs
• vhub_test_kaldi - default test version of our system deployed on our test extension, using KALDI ASR, logging locally into ../call_logs
• vhub_test_kaldi_nfs - default test version of our system deployed on our test extension, using KALDI ASR and logging to NFS

Running the system without the full UFAL access¶

Users outside UFAL can run the system using the following commands:

• vhub_private_ext_google_only - default version of our system deployed on private extension specified in private_ext.cfg, using Google ASR, TTS, Directions, and KALDI ASR, logging locally into ../call_logs
• vhub_private_ext_google_kaldi - default version of our system deployed on private extension specified in private_ext.cfg, using Google TTS, Directions, and KALDI ASR, logging locally into ../call_logs

If you want to test the system on your private extension, then modify the private_ext.cfg config. You must set your SIP domain including the port, user login, and password (You can obtain a free extension at http://www.sipgate.co.uk). Please make sure that you do not commit your login information into the repository.

config = {
        'VoipIO': {
                # default testing extesion
                'domain':   "*:5066",
                'user':     "*",
                'password': "*",
        },
}

Also, you will have to create a “private” directory where you can store your private configurations. As the private default configuration is not part of the Git repository, please make your own empty version of the private default configuration as follows.

mkdir alex/resources/private
echo "config = {}" > alex/resources/private/default.cfg

Definition of dialogue acts¶

In a spoken dialogue system, the observations and the system actions are represented by dialogue acts. Dialogue acts represent basic intents (such as inform, request, etc.) and the semantic content in the input utterance (e.g. type=hotel, area=east). In some cases, the value can be omitted, for example, where the intention is to query the value of a slot e.g. request(food).

In the UFAL Dialogue Act Scheme (UDAS), a dialogue act (DA) is composed of one or more dialogue act items (DAI). A dialogue act item is defined as a tuple composed of a dialogue act type, a slot name, and the slot value. Slot names and slot values are domain dependent, therefore they can be many. In the examples which follows, the names of the slots and their values are drown form a information seeking application about restaurants, bars and hotels. For example in a tourist information domain, the slots can include “food” or “pricerange” and the values can be such as “Italian”, “Indian” or ”cheap”, “midpriced”, or “expensive”.

This can be described in more formal way as follows:

DA = (DAI)+
DAI = (DAT, SN, SV)
DAT = (ack, affirm, apology, bye, canthearyou, confirm,
    iconfirm, deny, hangup, hello, help, inform, negate,
    notunderstood, null, repeat, reqalts, reqmore, request,
    restart, select, thankyou)

where SN denotes a slot name and SV denotes a slot value.

The idea of dialogue comes from the information state update (ISU) approach of defining a dialogue state. In ISU, a dialogue act is understood as a set of deterministic operations on a dialogue state which which result in a new updated state. In the UFAL dialogue act scheme, the update is performed on the slot level.

The following explains each dialogue act type:

ack         - "Ok" - back channel
affirm      - simple "Yes"
apology     - apology for misunderstanding
bye         - end of a dialogue – simple "Goodbye"
confirm     - user tries to confirm some information
canthearyou - system or user does not hear the other party
deny        - user denies some information
hangup      - the user hangs up
hello       - start of a dialogue – simple "Hi"
help        - request for help
inform      - user provides some information or constraint
negate      - simple "No"
null        - silence, empty sentence, something that is not possible to
                interpret, does nothing

null It can be also used when converting a dialogue act item confusion network into an N-best list to hold all the probability mass connected with all dialogue acts which were not added to the N-best list. In other words probability mass of pruned DA hypotheses.

notunderstood - informs that the last input was not understood
repeat      - request to repeat the last utterance
irepeat     - repeats the last utterance
reqalts     - ask for alternatives
reqmore     - ask for more details
request     - user requests some information
restart     - request a restart of the dialogue
select      - user or the system wants the other party to select between
                two values for one slot
thankyou    - simple "Thank you"

NOTE: Having this set of acts we cannot confirm that something is not equal to something, e.g. confirm(x!=y) → confirm(pricerange != 'cheap') → “Isn’t it cheap?” If we used confirm(pricerange = 'cheap') then it means “Is it cheap?” In both cases, it is appropriate to react in the same way e.g. inform(pricerange='cheap') or deny(pricerange = 'cheap').

NOTE: Please note that all slot values are always placed in quotes ".

Dialogue act examples¶

This section presents examples of dialogue acts:

ack()                           'ok give me that one'
                                'ok great'

affirm()                        'correct'
                                'erm yeah'

appology()                      'sorry'
                                'sorry I did not get that'

bye()                           'allright bye'
                                'allright then bye'

canthearyou()                   'hallo'
                                'are you still there'

confirm(addr='main square')     'erm is that near the central the main square'
                                'is it on main square'

iconfirm(addr='main square')    'Ack, on main square,'


iconfirm(near='cinema')         'You want something near cinema'

deny(name='youth hostel')       'not the youth hostel'

deny(near='cinema')             'ok it doesn't have to be near the cinema'

hello()                         'hello'
                                'hi'
                                'hiya please'

help()                          'can you help me'


inform(='main square')          'main square'

inform(addr='dontcare')         'i don't mind the address'


inform(food='chinese')          'chinese'
                                'chinese food'
                                'do you have chinese food'

negate()                        'erm erm no i didn't say anything'
                                'neither'
                                'no'

null()                          '' (empty sentence)
                                'abraka dabra' (something not interpretable)

repeat()                        'can you repeat'
                                'could you repeat that'
                                'could you repeat that please'

reqalts()                       'and anything else'
                                'are there any other options'
                                'are there any others'

reqmore()                       'can you give me more dtails'

request(food)                   'do you know what food it serves'
                                'what food does it serve'

request(music)                  'and what sort of music would it play'
                                'and what type of music do they play in these bars'

restart()                       'can we start again please'
                                'could we start again'

select(food="Chinese")&select(food="Italian)
                                'do you want Chinese or Italian food'

thankyou()                      'allright thank you then i'll have to look somewhere else'
                                'erm great thank you'

If the system wants to inform that no venue is matching provided constraints, e.g. “There is no Chinese restaurant in a cheap price range in the city centre” the system uses the inform(name='none') dialogue acts as in

Utterance: There is no Chinese restaurant in a cheap price range in the city centre”

Dialogue act: inform(name='none')&inform(venue_type='restaurant')&inform(food_type='Chinese')&inform(price_range='cheap')

There are examples of dialogue acts composed of several DAIs:

reqalts()&thankyou()            'no thank you somewhere else please'

request(price)&thankyou()       'thank you and how much does it cost'
                                'thank you could you tell me the cost'

affirm()&inform(area='south')&inform(music='jazz')&inform(type='bar')&request(name)
                                'yes i'd like to know the name of the bar in the south part of town that plays jazz music'
                                'yes please can you give me the name of the bar in the south part of town that plays jazz music'

confirm(area='central')&inform(name='cinema')
                                'is the cinema near the centre of town'


deny(music='pop')&inform(music='folk')
                                'erm i don't want pop music i want folk folk music'


hello()&inform(area='east')&inform(drinks='cocktails')&inform(near='park')&inform(pricerange='dontcare')&inform(type='hotel')
                                'hi i'd like a hotel in the east of town by the park the price doesn't matter but i'd like to be able to order cocktails'

An example dialogue form tourist information domain is in the following table:

Semantic Decoding and Ambiguity¶

Very often there are many ways as to map (to interpret) a natural utterance into a dialogue act, , some times because of natural ambiguity of a sentence – sometimes because of the speech recognition errors. Therefore, a semantic parser will generate multiple hypotheses. In this case, each hypothesis will be assigned a probability meaning the likelihood of being correct and the dialogue manager will resolve this ambiguity in the context of the dialogue (e.g. other sentences).

For example, the utterance “I wan an Italian restaurant erm no Indian” can be interpreted as:

inform(venue="restaurant")&inform(food="Italian")&deny(food=Indian)

or:

inform(venue="restaurant")&inform(food="Indian")

In the first case, the utterance is interpreted that the user wants Italian restaurant and does not want Indian. However, in the second case, the user corrected what he just mistakenly said (that he wants Indian restaurant).

Please remember that semantic parsers should interpret an utterance only on the information present in the sentence. It is up to the dialogue manager to interpret it in the context of the whole dialogue:

inform(type=restaurant)&inform(food='Chinese')
'I want a Chinese restaurant'

inform(food='Chinese')
'I would like some Chinese food'

In the first case, the user explicitly says that he/she is looking for a restaurant. However, in the second case, the user said that he/she is looking for some venue serving Indian food which can be both a restaurant or only a take-away.

Building a statistical SLU parser for a new domain¶

From experience, it appears that the easiest approach to build a statistical parser for a new domain is to start with build a handcrafted (rule based) parser. There are several practical reasons for that:

1. a handcrafted parser can serve as a prototype module for a dialogue system when no data is available,
2. a handcrafted parser can serve as a baseline for testing data driven parsers,
3. a handcrafted parser in information seeking applications, if well implemented, achieves about 95% accuracy on transcribed speech, which is close to accuracy of what the human annotators achieve,
4. a handcrafted parser can be used to obtain automatic SLU annotation which can be later hand corrected by humans.

To build a data driven SLU, the following approach is recommended:

1. after some data is collected, e.g. a prototype of dialogue system using a handcrafted parser, the audio from the collected calls is manually transcribed and then parsed using the handcrafted parser,
2. the advantage of using automatic SLU annotations is that they are easy to obtain and reasonably accurate only several percent lower to what one can get from human annotators.
3. if better accuracy is needed then it is better to fix the automatic semantic annotation by humans,
4. then a data driven parser is trained using this annotation

Note that the main benefit of data driven SLU methods comes from the ability to robustly handle erroneous input. Therefore, the data driven SLU should be trained to map the recognised speech to the dialogue acts (e.g. obtained by the handcrafted parser on the transcribed speech and then corrected by human annotator).

Comments¶

The previous sections described the general set of dialogue acts in UFAL dialogue systems. However, exact set of dialogue acts depends on a specific application domain and is defined by the domain specific semantic parser.

The only requirement is that all the output of a parser must be accepted by the dialogue manager developed for the particular domain.

Apendix A: UFAL Dialogue acts¶

Resources used by public transport direction finders and weather service¶

The sources of the data that are loaded by the application are:

• cities.expanded.txt – list of known cities and towns in the USA. (tab-separated: slot value name + possible forms separated by semicolons; lines starting with ‘#’ are ignored)
• states.expanded.txt – list of us state names (same format).
• stops.expanded.txt – list of known stop names (same format) in NY.
• stops.expanded.txt – list of known stop names (same format) in NY.
• streets.expanded.txt – list of known street names (same format)
• boroughs.expanded.txt – list of known borough names (same format)
• cities.locations.csv – tab separated list of known cities and towns, their state and geo location (longitude|latitude).
• stops.locations.csv – tab separated list of stops, their cities and geo location (longitude|latitude).
• stops.borough.locations.csv – tab separated list of stops, their boroughs and geo location (longitude|latitude).
• streets.types.locations.csv – tab separated list of streets, their boroughs and type (Avenue, Street, Court etc.)

All of these files are generated from states-in.csv, cities-in.csv, stops-in.csv, streets-in.csv and boroughs-in.csv located at ./preprocessing/resources using the expand_states_script.py, expand_cities_script.py, expand_stops_script.py, expand_streets_script.py and expand_boroughs_script.py script respectively. Please note that all forms in *.expanded.txt files are lowercased and do not include any punctuation.

Colloquial name variants that are added by hand are located in the ./preprocessing/resources/*-add.txt files for each slot and are appended to the expansion process.

build_data.sh script is combining all the expansion scripts mentioned earlier into one process.

Available data¶

At this moment, we only have data which were automatically generated using our handcrafted SLU (HDC SLU) parser on the transcribed audio. In general, the quality of the automatic annotation is very good.

The data can be prepared using the prapare_data.py script. It assumes that there exist the indomain_data directory with links to directories containing asr_transcribed.xml files. Then it uses these files to extract transcriptions and generate automatic SLU annotations using the PTIENHDCSLU parser from the hdc_slu.py file.

The script generates the following files:

• *.trn: contains manual transcriptions
• *.trn.hdc.sem: contains automatic annotation from transcriptions using handcrafted SLU
• *.asr: contains ASR 1-best results
• *.asr.hdc.sem: contains automatic annotation from 1-best ASR using handcrafted SLU
• *.nbl: contains ASR N-best results
• *.nbl.hdc.sem: contains automatic annotation from n-best ASR using handcrafted SLU

The script accepts --uniq parameter for fast generation of unique HDC SLU annotations. This is useful when tuning the HDC SLU.

The script also accepts --fast parameter for fast approximate preparation of all data. It approximates the HDC SLU output from an N-best list using output obtained by parsing the 1-best ASR result.

Building the models¶

First, prepare the data. Link the directories with the in-domain data into the indomain_data directory. Then run the following command:

./prepare_data.py

Second, train and test the models.

./train.py && ./test.py && ./test_bootstrap.py

Third, look at the *.score files or compute the interesting scores by running:

./print_scores.sh

Future work¶

• The prepare_data.py will have to use ASR, NBLIST, and CONFNET data generated by the latest ASR system instead of the logged ASR results because the ASR can change over time.
• Condition the SLU DialogueActItem decoding on the previous system dialogue act.

Evaluation¶

Please note that the scoring is implicitly ignoring all non-speech events.

Ref: all.trn
Tst: all.asr
|==============================================================================================|
|            | # Sentences  |  # Words  |   Corr   |   Sub    |   Del    |   Ins    |   Err    |
|----------------------------------------------------------------------------------------------|
| Sum/Avg    |     9111     |   24728   |  56.15   |  16.07   |  27.77   |   1.44   |  45.28   |
|==============================================================================================|

ASR¶

The ASR can generate either:

• a valid utterance
• the ```` empty sentence word to denote that the input was silence
• the _noise_ word to denote that the input was some noise or other sound which is not a regular word
• the _laugh_ word to denote that the input was laugh
• the _ehm_hmm_ word to denote that the input was ehm or ehm sounds
• the _inhale_ word to denote that the input was inhale sound
• the _other_ word to denote that the input was something else that was lost during speech processing approximations such as N-best list enumeration or when the ASR did not provided any result. This is because we do not know what the input was and it can be both something important or worth ignoring. As such, it deserves special treatment in the system.

SLU¶

The SLU can generate either:

• a ordinary dialogue act
• the null() act which should be ignored by the DM, and the system should respond with silence()
• the silence() act which denote that the user was silent, a probably reasonable system response is silence() as well
• the other() act which denote that the input was something else that was lost during processing

The SLU should map:

• ```` to silence() - silence will be processed in the DM
• _noise_, _laugh_, _ehm_hmm_, and _inhale_ to null() - noise can be ignored in general
• _other_ to other() - other hypotheses will be handled by the DM, mostly by responding “I did not get that. Can you ... ?”

DM¶

The DM can generate either:

• a normal dialogue act
• the silence() dialogue act

The DM should map:

• null() to silence() - because the null() act denote that the input should be ignored; however there is a problem with this, read the note below for current workaround for this
• silence() to silence() or a normal dialogue act - the DM should be silent or to ask the user “Are still there?”
• other() to notunderstood() - to show the user that we did not understood the input and that the input should be rephrased instead of just being repeated.

PROBLEM As of now, both handcrafted and trained SLUs cannot correctly classify the other() dialogue act. It has a very low recall for this DA. Instead of the other() DA it returns the null() DA. Therefore, the null() act is processed in DMs as if it was the other() DA for now.

Description¶

This application provides information about public transport connections in New York using English language. Just say origin and destination stops and the application will find and tell you about the available connections. You can also specify a departure or arrival time if necessary. It offers bus, tram and metro city connections, and bus and train inter-city connections.

The application is available at the telephone number 1-855-528-7350.

You can also:

• ask for help
• ask for a “restart” of the dialogue and start the conversation again
• end the call - for example, by saying “Good bye.”
• ask for repetition of the last sentence
• confirm or reject questions
• ask about the departure or destination station, or confirm it
• ask for the number of transits
• ask for the departure or arrival time
• ask for an alternative connection
• ask for a repetition of the previous connection, the first connection, the second connection, etc.

In addition, the application provides also information about:

• weather forecast
• current time

Description¶

This application provides information about public transport connections in Czech Republic using the Czech language. Just say (in Czech) your origin and destination stops and the application will find and tell you about the available connections. You can also specify a departure or arrival time if necessary. It offers bus, tram and metro city connections, and bus and train inter-city connections.

The application is available at the toll-free telephone number +420 800 899 998.

You can also:

• ask for help
• ask for a “restart” of the dialogue and start the conversation again
• end the call - for example, by saying “Good bye.”
• ask for repetition of the last sentence
• confirm or reject questions
• ask about the departure or destination station, or confirm it
• ask for the number of transits
• ask for the departure or arrival time
• ask for an alternative connection
• ask for a repetition of the previous connection, the first connection, the second connection, etc.

In addition, the application provides also information about:

• weather forecast
• the current time

Representation of semantics¶

Suggestion (MK): It would be better to treat the specification of hours and minutes separately. When they are put together, all ways the whole time expression can be said have to be enumerated in the CLDB manually.

Training models for a new language¶

Scripts for Czech and English are already created. If you need models for a new language, you can start by copying all the original scripts and renaming them so as to reflect the new language in their name (substitute _en or _cs with your new language code). You can do this by issuing the following command (we assume $OLDLANG is set to either en or cs and $NEWLANG to your new language code):

bash htk $ find . -name "*_$OLDLANG*" |
           xargs -n1 bash -c "cp -rvn \$1 \${1/_$OLDLANG/_$NEWLANG}" bash

Having done this, references to the new files’ names have to be updated, too:

bash htk $ find . -name "*_$NEWLANG*" -type f -execdir \
           sed --in-place s/_$OLDLANG/_$NEWLANG/g '{}' \;

Furthermore, you need to adjust language-specific resources to the new language in the following ways:

htk/model_voip_$NEWLANG/monophones0

List all the phones to be recognised, and the special sil phone.

htk/model_voip_$NEWLANG/monophones1

List all the phones to be recognised, and the special sil and sp phones.

htk/model_voip_$NEWLANG/tree_ques.hed

Specify phonetic questions to be used for building the decision tree for phone clustering (see [HTKBook], Section 10.5).

htk/bin/PhoneticTranscriptionCS.pl

You can start from this script or use a custom one. The goal is to implement the orthography-to-phonetics mapping to obtain sequences of phones from transcriptions you have.

htk/common/cmudict.0.7a and htk/common/cmudict.ext

This is an alternative approach to the previous point – instead of programming the orthography-to-phonetics mapping, you can list it explicitly in a pronouncing dictionary.

Depending on the way you want to implement the mapping, you want to set $OLDLANG to either cs or en.

To make the scripts work with your new files, you will have to update references to scripts you created. All scripts are stored in the htk/bin, htk/common, and htk directories as immediate children, so you can make the substitutions only in these files.

Credits and the licence¶

The scripts are based on the HTK Wall Street Journal Training Recipe written by Keith Vertanen (http://www.keithv.com/software/htk/). His code is released under the new BSD licence. The licence note is at http://www.keithv.com/software/htk/. As a result we can re-license the code under the APACHE 2.0 license.

The results¶

• total training data for voip_en is about 20 hours
• total training data for voip_cs is about 8 hours
• mixtures - there is 16 mixtures is slightly better than 8 mixtures for voip_en
• there is no significant difference in alignment of transcriptions with -t 150 and -t 250
• the Julius ASR performance is about the same as of HDecode
• HDecode works well when cross word phones are trained, however the - performance of HVite decreases significantly
• when only word internal triphones are trained then the HDecode works, - however, its performance is worse than the HVite with a bigram LM
• word internal triphones work well with Julius ASR, do not forget disable CCD (it does not need context handling - though it still uses triphones)
• there is not much gain using the trigram LM in the Caminfo domain (about 1%)

Resources used by public transport direction finders¶

The sources of the data that are loaded by the application are:

• cities.expanded.txt – list of known cities and towns in the Czech Rep. (tab-separated: slot value name + possible surface forms separated by semicolons; lines starting with ‘#’ are ignored)
• stops.expanded.txt – list of known stop names (same format)
• cities_stops.tsv – “compatibility table”: lists compatible city-stops pairs, one entry per line (city and stop are separated by tabs). Only the primary stop and city names are used here.

The files cities.expanded.txt and stops.expanded.txt are generated from cities.txt and stops.txt using the expand_stops.py script (see documentation in the file itself; you need to have Morphodita Python bindings installed to successfully run this script). Please note that the surface forms in them are lowercased and do not include any punctuation (this can be obtained by setting the -l and -p parameters of the expand_stops.py script).

Colloquial stop names’ variants that are added by hand are located in the stops-add.txt file and are appended to the stops.txt before performing the expansion.

Additional resources for the CRWS/IDOS directions finder¶

Since the CRWS/IDOS directions finder uses abbreviated stop names that need to be spelled out in ALEX, there is an additional resource file loaded by the system:

• idos_map.tsv – a mapping from the slot value names (city + stop) to abbreviated CRWS/IDOS names (stop list + stop)

The convert_idos_stops.py script is used to expand all possible abreviations and produce a mapping from/to the original CRWS/IDOS stop names as they appear, e.g., at the IDOS portal .

Resources used by the weather information service¶

The weather service uses one additional file:

• cities_locations.tsv – this file contains GPS locations of all cities in the Czech Republic.

Available data¶

At this moment, we only have data which were automatically generated using our handcrafted SLU (HDC SLU) parser on the transcribed audio. In general, the quality of the automatic annotation is very good.

The data can be prepared using the prapare_data.py script. It assumes that there exist the indomain_data directory with links to directories containing asr_transcribed.xml files. Then it uses these files to extract transcriptions and generate automatic SLU annotations using the PTICSHDCSLU parser from the hdc_slu.py file.

The script generates the following files:

• *.trn: contains manual transcriptions
• *.trn.hdc.sem: contains automatic annotation from transcriptions using handcrafted SLU
• *.asr: contains ASR 1-best results
• *.asr.hdc.sem: contains automatic annotation from 1-best ASR using handcrafted SLU
• *.nbl: contains ASR N-best results
• *.nbl.hdc.sem: contains automatic annotation from n-best ASR using handcrafted SLU

The script accepts --uniq parameter for fast generation of unique HDC SLU annotations. This is useful when tuning the HDC SLU.

Building the DAILogRegClassifier models¶

First, prepare the data. Link the directories with the in-domain data into the indomain_data directory. Then run the following command:

./prepare_data.py

Second, train and test the models.

::

cd ./dailogregclassifier

./train.py && ./test_trn.py && ./test_hdc.py && ./test_bootstrap_trn.py && ./test_bootsrap_hdc.py

Third, look at the *.score files or compute the interesting scores by running:

./print_scores.sh

Future work¶

• Exploit ASR Lattices instead of long NBLists.
• Condition the SLU DialogueActItem decoding on the previous system dialogue act.

Evaluation¶

Please note that the scoring is implicitly ignoring all non-speech events.

Ref: all.trn
Tst: all.asr
|==============================================================================================|
|            | # Sentences  |  # Words  |   Corr   |   Sub    |   Del    |   Ins    |   Err    |
|----------------------------------------------------------------------------------------------|
| Sum/Avg    |     9111     |   24728   |  56.15   |  16.07   |  27.77   |   1.44   |  45.28   |
|==============================================================================================|

Interactive tests¶

This directory contains only (interactive) tests, which can’t be automated and the results must be verified by humans! E.g. playing or recording audio, testing VOIP connections.

Unit tests¶

Note that the unit tests should be placed in the same directory as the tested module and the name should be test_*.py e.g. test_module_name.py.

Using unittest module:

$ python -m unittest alex.test.test_string

This approach works everywhere but doesn’t support test discovery.

Using nose test discovery framework, testing can largely automated. Nose searchs through packages and runs every test. Tests must be named test_<something>.py and must not be executable. Tests doesn’t have to be run from project root, nose is able to find project root on its own.

How should my unit tests look like?

• Use unittest module
• Name the test file test_<something>.py
• Make the test file not executable

Data for building general LMs¶

To get free general out-of-domain text data, we use the free W2C – Web to Corpus – Corpora available from the LINDAT project at: https://ufal-point.mff.cuni.cz/repository/xmlui/handle/11858/00-097C-0000-0022-6133-9

• English: https://ufal-point.mff.cuni.cz/repository/xmlui/bitstream/handle/11858/00-097C-0000-0022-6133-9/eng.txt.gz
• Czech: https://ufal-point.mff.cuni.cz/repository/xmlui/bitstream/handle/11858/00-097C-0000-0022-6133-9/ces.txt.gz

Structure of each domain scripts¶

Each of the projects should contain:

1. build.py - builds the final LMs, and computes perplexity of final LMs

Necessary files for the LM¶

For each domain the LM package should contain:

1. ARPA trigram language model (final.tg.arpa)
2. ARPA bigram language model (final.bg.arpa)
3. HTK wordnet bigram language model (final.bg.wdnet)
4. List of all words in the language model (final.vocab)
5. Dictionary including all words in the language model using compatible phone set with the language specific acoustic model (final.dict - without pauses and final.dict.sp_sil with short and long pauses)

CamInfoRest¶

For more details please see alex.applications.CamInfoRest.lm.README.

PTIcs¶

For more details please see Building the language model for the Public Transport Info telephone service (Czech).

Summary¶

• Requires KALDI installation and Linux environment. (Tested on Ubuntu 10.04, 12.04 and 12.10.) Note: We recommend Kaldi fork Pykaldi, because you will need it also for integrated Kaldi decoder to Alex.
• Recipes deployed with the Kaldi toolkit are located at $KALDI_ROOT/egs/name_of_recipe/s[1-5]/. This recipe requires to set up $KALDI_ROOT variable so it can use Kaldi binaries and scripts from $KALDI_ROOT/egs/wsj/s5/.

Details¶

• The recommended settings are stored at env_LANG_RCONG.sh e.g env_voip_en.sh
• We recommend to adjust the settings in file env_LANG_RCONG_CUSTOM.sh` e.g. env_voip_en_CUSTOM.sh. See below. Do not commit this file to the git repository!
• Our scripts prepare the data to the expected format to $WORK directory.
• Experiment files are stored to $EXP directory.
• The symbolic links to $KALDI_ROOT/wsj/s5/utils and $KALDI_ROOT/wsj/s5/steps are automatically created.
• The files path.sh, cmd.sh are necessary to utils and steps scripts. Do not relocate them!
• Language model (LM) is either built from the training data using SRILM or specified in env_LANG_RCOND.sh.

Example of env_voip_en_CUSTOM.sh

# uses every utterance for the recipe every_N=10 is nice for debugging
export EVERY_N=1
# path to built Kaldi library and scripts
export KALDI_ROOT=/net/projects/vystadial/lib/kronos/pykaldi/kaldi

export DATA_ROOT=/net/projects/vystadial/data/asr/cs/voip/
export LM_paths="build0 $DATA_ROOT/arpa_bigram"
export LM_names="build0 vystadialbigram"

export CUDA_VISIBLE_DEVICES=0  # only card 0 (Tesla on Kronos) will be used for DNN training

Running experiments¶

Before running the experiments, check that:

• you have the Kaldi toolkit compiled: - http://github.com/UFAL-DSG/pykaldi (Recommended Kaldi fork, tested, necessary for further Alex integration) - http://sourceforge.net/projects/kaldi/ (alternative, main Kaldi repository) - In order to compile Kaldi we suggest:

# build openfst
pushd kaldi/tools
make openfst_tgt
popd

# download ATLAS headers
pushd kaldi/tools
make atlas
popd

# generate Kaldi makefile ``kaldi.mk`` and compile Kaldi
pushd kaldi/src
./configure
make && make test
popd

• you have SRILM compiled. (This is needed for building a language model) unless you supply your own LM in the ARPA format.)

pushd kaldi/tools
# download the srilm.tgz archive from http://www.speech.sri.com/projects/srilm/download.html
./install_srilm.sh
pushd

• the train_LANG_RCOND script will see the Kaldi scripts and binaries. Check for example that $KALDI_ROOT/egs/wsj/s5/utils/parse_options.sh is valid path.
• in cmd.sh, you switched to run the training on a SGE[*] grid if required (disabled by default) and njobs is less than number of your CPU cores.

Start the recipe by running bash train_LANG_RCOND.sh.

Extracting the results and trained models¶

The main script, bash train_LANG_RCOND.sh, performs not only training of the acoustic models, but also decoding. The acoustic models are evaluated during running the scripts and evaluation reports are printed to the standard output.

The local/results.py exp command extracts the results from the $EXP directory. It is invoked at the end of the train_LANG_RCOND.sh script.

If you want to use the trained acoustic model outside the prepared script, you need to build the HCLG decoding graph yourself. (See http://kaldi.sourceforge.net/graph.html for general introduction to the FST framework in Kaldi.) The HCLG.fst decoding graph is created by utils/mkgraph.sh. See run.sh for details.

Credits and license¶

The scripts were based on Voxforge KALDI recipe http://vpanayotov.blogspot.cz/2012/07/voxforge-scripts-for-kaldi.html . The original scripts as well as theses scripts are licensed under APACHE 2.0 license.

Experiments and the notes for the NN VAD¶

• testing is performed on randomly sampled data points (20%) from the entire set
• L2 regularisation must be very small, in addition it does not help much
• instead of MFCC, we use mel-filter banks coefficients only. It looks like the performance is the same or even better
• as of 2014-09-19 the best compromise between the model complexity and the performance appears to be.
  • 30 previous frames
  • 15 next frames
  • 512 hidden units
  • 4 hidden layers
  • tanh hidden layer activation
  • 4x amplification of the central frame compared to outer frames
  • discriminative pre-training
  • given this setup we get about 95.3 % frame accuracy on about 27 million of all data

Data¶

data_vad_sil    # a directory with only silence, noise data and its mlf file
data_voip_cs    # a directory where CS data reside and its MLF (phoneme alignment)
data_voip_en    # a directory where EN data reside and its MLF (phoneme alignment)
model_voip      # a directory where all the resulting models are stored.

Scripts¶

upload_models.sh                     # uploads all available models in ``model_voip`` onto the Alex online update server
train_voip_nn_theano_sds_mfcc.py     # this is the main trainign script, see its help for more details
bulk_train_nn_theano_mbo_31M_sgd.sh  # script with curently ``optimal`` setting for VAD

Comments¶

To save some time especially for multiple experiments on the same data, we store preprocessed speech parametrisation. The speech parametrisation is stored because it takes about 7 hours to produce. However, it takes only 1 minute to load from a disk file. The model_voip directory stores this speech parametrisation in *.npc files. There fore if new data is added, then these NPC files must be deleted. If there are no NPC files then they are automatically generated from the available WAV files.

The data_voip_{cs,en} alignment files (mlf files) can be trained using scripts alex/alex/tools/htk or alex/alex/tools/kaldi. See the train_voip_{cs,en}.sh scripts in one of the directories. Note that the Kaldi scripts first store alignment in ctm format and later converts it to mlf format.

Summary¶

The build_hclg.sh script formats language model (LM) and acoustic model (AM) into files (e.g. HCLG) formated for Kaldi decoders.

The scripts extracts phone lists and sets from lexicon given the acoustic model (AM), the phonetic decision tree (tree) and the phonetic dictionary(lexicon).

The script silently supposes the same phone lists are generated from lexicon as the these used for training AM. If they are not the same, the script crashes.

The use case. Run the script with trained AM on full phonetic set for given language, pass the script also the tree used for tying the phonetic set and also give the script your LM and corresponding lexicon. The lexicon and the LM should also cover the full phonetic set for given language.

The decode_indomain.py script uses HCLG.fst and the rest of files generated by build_hclg.sh and performes decoding on prerecorded wav files. The reference speech transcription and path to the wav files are extracted from collected call logs. The wav files should be from one domain and the LM used to build HCLG.fst should be from the same domain. The decode_indomain.py also evaluates the decoded transcriptions. The Word Error Rate (WER), Real Time Factor (RTF) and other minor statistics are collected.

Dependencies of build_hclg.sh¶

The build_hclg.sh script requires the scripts listed belofw from $KALDI_ROOT/egs/wsj/s5/utils. The “utils scripts transitevely uses scripts from $KALDI_ROOT/egs/wsj/s5/steps. The dependency is solved in path.sh script which create corresponding symlinks and adds Kaldi binaries to your system path.

You just needed to set up KALDI_ROOT root variable and provide correct arguments. Try to run

Needed scripts from utils symlinked directory. * gen_topo.pl * add_lex_disambig.pl * apply_map.pl * eps2disambig.pl * find_arpa_oovs.pl * gen_topo.pl * make_lexicon_fst.pl * remove_oovs.pl * s2eps.pl * sym2int.pl * validate_dict_dir.pl * validate_lang.pl * parse_options.sh

Scripts from the list use Kaldi binaries, so you need Kaldi compiled on your system. The script path.sh adds Kaldi binaries to the PATH and also creates symlinks to utils and steps directories, where the helper scripts are located. You only need to set up $KALDI_ROOT variable.

Running the system at UFAL with the full UFAL access¶

There are multiple configuration that can used to run the system. In general, it depends on what components you want go use and on what telephone extension you want to run the system.

Within UFAL, we run the system using the following commands:

• vhub_mta1 - deployment of our live system on a 1-855-528-7350 phone number, with the default configuration
• vhub_mta2 - a system deployed to backup the system above
• vhub_mta3 - a system deployed to backup the system above
• vhub_mta_btn - a system deployed to backup the system above accessible via web page http://alex-ptien.com

To test the system we use:

• vhub_devel - default devel version of our system deployed on our test extension, logging locally into ../call_logs

Running the system without the full UFAL access¶

Users outside UFAL can run the system using the following commands:

• vhub_private_ext_google_only_hdc_slu - default version of our system deployed on private extension specified in private_ext.cfg, using HDC_SLU, Google ASR, TTS, Directions, logging locally into ../call_logs
• vhub_private_ext_google_kaldi_hdc_slu - default version of our system deployed on private extension specified in private_ext.cfg, using HDC_SLU, Google TTS, Directions, and KALDI ASR, logging locally into ../call_logs

If you want to test the system on your private extension, then modify the private_ext.cfg config. You must set your SIP domain including the port, user login, and password. Please make sure that you do not commit your login information into the repository.

config = {
        'VoipIO': {
                # default testing extesion
                'domain':   "*:5066",
                'user':     "*",
                'password': "*",
        },
}

Also, you will have to create a “private” directory where you can store your private configurations. As the private default configuration is not part of the Git repository, please make your own empty version of the private default configuration as follows.

mkdir alex/resources/private
echo "config = {}" > alex/resources/private/default.cfg