Experiments with baseline system

Experiments with the Baseline System

[STAT MT HOME] [SHARED TASK 07] | RESULTS 07] | [BASELINE SYSTEM 07] | [PROCEEDINGS] | [PROGRAM]

Using the open source Moses it is possible to build a baseline system that is on par with the best submission from the '06 workshop. Step-by-step instructions are given here. Here are the ideas connected with some basic components.

Linguistic Data mining

Mine the phrase and word alignment models for information about target and source languages and their relations.
Data can be extracted from these models with very simple scripts, since they all exist online as text files.
For example, one can produce a crude French-oriented "sense model" of the meanings for break with a simple "grep" command.
Data is bidrectional, so one can follow links back and forth to look for supporting the data or widen the search.
One can do this for many basic verbs looking for "sense patterns."

Preprocessing

The data ineuroparl is preprocessed in certain ways to facilitate learning of the translation model. For example, long sentences are thrown out. The words are all lower-cased. There are good justifications for both of these moves.
Experiment with opther forms of preprocessing that might promote better alignment models.

Giza and mkclas (incorporated into moses)

The basic word alignment models implemented in giza++ are described in A Systematic Comparison of Various Statistical Aligment Models, Franz Josef Och and Hermann Ney, 2003, Computational Linguistics 29.1, pp. 19--51.
The basic algorithm used by mkcls is described in "An efficient Method for Determining Bilingual Word Classes", 1999, Franz Jozef Och, EACL pp. 71-76. (ps, pdf)
Word reordering: [= distortion model]

Phrase Model (interventions)

Phrase models: The Pharoah system accepted XML annotations that forced certain translations for phrases (this was motivated by problems such as the translations of named entities)
Er erzielte <NUMBER english='17.55'>17.55</NUMBER> Punkte .
Probabilities could also be given, for example:
Es ist <NPPP english='a small house|a little house' prob='0.6|0.4'> ein kleines Haus

Implement a phrasicon for moses by messing with the phrase translation table. But can you do this is a principled way?
Implement some version of word-reordering for a language where it might clearly help, like German. Test and evaluate.

Tuning experiments: Fiddle

Use the moses.ini created by the moses train-factored-phrase-model script (in /scratch/mt/model/moses.ini) for the devtest.
Evaluate. What happens?
Try setting new weights by hand. Evaluate. What happens?
How dependent are the tuning paramters arrived at by the MER algorithm on the development test set? Can you test this without hunting for lots of new data?

Substitute your own classes

Look at the class files in /scratch/mt/model/. Substitute your own versions.

Create your compact version

Statistical MT models are huge.
Running moses without filtering the model will cause a failure on ngram, morpheme (our 2GB RAM machines) while loading the model into memory (so you have the pleasant experience of crashing moses while trying to translate je parle francais) Just before failure we have a 3GB virtual image using 1.4 GB swap (according to top). Then:
[gawron@morpheme evaluation]$ ~/mt/moses/moses-cmd/src/moses -config /scratch/mt/tuning/moses.weight-reused.ini -input-file ~/new_test.fr >> ~/new_test.en Defined parameters (per moses.ini or switch): config: /scratch/mt/tuning/moses.weight-reused.ini distortion-file: 0-0 msd-bidirectional-fe 6 /scratch/mt/model/reordering-table.msd-bidirectional-fe.0.5.0-0.gz distortion-limit: 6 input-factors: 0 input-file: /home/gawron/new_test.fr lmodel-file: 0 0 5 /scratch/mt/lm/europarl.lm mapping: 0 T 0 ttable-file: 0 0 5 /scratch/mt/model/phrase-table.0-0.gz ttable-limit: 20 0 weight-d: 0.062701 0.147073 -0.013620 -0.034194 0.155787 0.054588 0.135913 weight-l: 0.086438 weight-t: 0.042401 0.049556 0.058195 0.033184 0.036851 weight-w: -0.089499 Loading lexical distortion models... have 1 models Creating lexical reordering... weights: 0.147 -0.014 -0.034 0.156 0.055 0.136 Loading table into memory...done. Created lexical orientation reordering Start loading LanguageModel /scratch/mt/lm/europarl.lm : [416.000] seconds moses: ../../include/LHash.cc:133: void LHash::alloc(unsigned int) [with KeyT = unsigned int, DataT = float]: Assertion `body != 0' failed. Aborted [gawron@morpheme evaluation]$
Filtering the model is data specific. If you filter the model on the devtest dataset, francais becomes an unknown word in Je parle francais. If you slip francais into the dataset somehow, it does not.
Filtering is done as follows:
/opt/bin/moses-scripts/scripts-20070526-0023/training/filter-model-given-input.pl working-dir/evaluation/filtered.devtest2006 working-dir/tuning/moses.weight-reused.ini [file with input data to filter]
A new model is created which discards all vocab and phrases not in the input dataset.
Experiment with different ways of filtering the models withOUT looking at a specific dataset. Looking at the filtered dataset created for the devtest will give you some idea of a practical goal.

Create your own MT system

Get yourself a parallel corpus in some language you know. We have several. See also the parallel corpora section here. Train and build a new system along the lines of the baseline model.
Build systems in both directions. Compare the two systems. Use bleu scores and your own judgments to evaluate the two systems.
Explore the reasons for any differences in performance. Write it up.
Important: dont forget to set aside and prepare development (for tuning!) and devtest data (for testing!).

Try out a heuristic word-alignment scheme

Analyze the evaluation results in WMT07

One system scored consistently high with human judges and consistently low with automatic evaluation measure slike the BLEU score. That system is Systrans which, interestingly, is not a statistical MT system. This suggests interesting possibilities. Explore them by exploring the actual submitted system data and the human judge data here.
German-English is significantly easier than English-German, at least according to WMT07. Hypothesize why this might be. Explore the hypothesis in the evaluation results for WMT07. Test your hypothesis.
Analyze the translation of a set of problematic constructions, such as the passive. What is going wrong and why? Test your hypothesis with the baseline system. Could word-reordering rules help? Test this idea.