[ Skip to the content ]

Institute of Formal and Applied Linguistics Wiki


[ Back to the navigation ]

This is an old revision of the document!


Table of Contents

Introduction

Article is about parallel sentence extraction from Wikipedia. This resource can be viewed as comparable corpus in which the document alignment is already provided by the interwiki links.

Training models

Authors train three models:

When the binary classifier is used, there is a substantial class imbalance: O(n) positive examples and O(n²) negative examples.

The ranking model selects either a sentence in the target document or 'null' for each sentence target in the source document. This way there is no problem of class imbalance issue of the binary classifier.

A conditional random field is a type of discriminative undirected probabilistic graphical model. It is most often used for labeling or parsing of sequential data, such as natural language text.

Features

Category 1: Features derived from word alignment

  1. Číslovaný seznam log probability of the alignment

[ Back to the navigation ] [ Back to the content ]