Interview with Nikos on Cross-Language Information Retrieval

university-of-limerick-logo

Nikos Katris, submitted his thesis; ‘Evaluation of Two Statistical Machine Translation Systems within a Greek-English Cross-Language Information Retrieval Architecture’ to University of Limerick  in October 2015. In his research he compared the results of KantanMT with the Moses system for information retrieval.

Nikos was supervised by Dr Richard Sutcliffe at the University of Limerick’s College of Science and Engineering Department of Computer Science and Information Systems (CSIS). Nikos kindly agreed to discuss his research in an interview. The University of Limerick and the Localisation Research Centre are KantanMT’s academic partners. Continue reading

Student Speak: Student at UCL Chats with KantanMT Team

architecture-1122359_1920Dissemination of Machine Translation innovation is a major priority for us at KantanMT. We believe that Academic Partnerships have a huge role to play in furthering the scope of research and innovation in the field of Machine Translation, and as such we have partnered with a number of Universities to help students use the KanataMT platform in a real word scenario.

We are always looking for ways to improve the KantanMT platform, and to keep our finger on the pulse of the KantanMT user experience, we asked one of the students using the platform to answer some questions about the platform.

Continue reading

Using F-Measure in Kantan BuildAnalytics

What is F-Measure ?

KantanMT Logo 800x800 F-Measure is an automated measurement that determines the precision and recall  capabilities of a KantanMT engine. F-Measure measures enables you to determine the  quality and performance of your KantanMT engine

  • To see the accuracy and performance of your engine click on the ‘F-measure Scores’ tab. You will now be directed to the ‘F-measure Scores’ page.

F-Measure tab

  • Place your cursor on the ‘F-measure Scores Chart’ to see the individual score of each segment. A pop-up will now appear on your screen with details of the segment under these headings, ‘Segment no.’, ‘Score’, ‘Source’, ‘Reference/Target’ and ‘KantanMT Output’.

Segment

  • To see the ‘F-measure Scores’ of each segment in a table format scroll down. You will now see a table with the headings ‘No’, ‘Source’, ‘Reference/Target’, ‘KantanMT Output’ and ‘Score’.
  • To see an even more in depth breakdown of a particular ‘Segment’ click on the Triangle beside the number of the segment you wish to view.Triangle
  • To reuse the engine as Test Data click on the ‘Reuse as Test Data’. When you do so, the ‘Reuse as Test Data’ button will change to ‘Delete Test Data’.Test Data
    Delete Test Data
  • To download the ‘F-measure Scores’, ‘BLEU Score’ and ‘TER Scores’ of all segments click on the ‘Download’ button on either the ‘F-measure Scores’, ‘BLEU Score’ or ‘TER Scores’ page.download

This is one of the features provided by Kantan BuildAnalytics to improve an engine’s quality after its initial training .To see other features used by Kantan BuildAnalytics please click on the link below .To get more information about KantanMT and the services we provide please contact our support team at  at info@kantanmt.com.

What is KantanBuildAnalytics™?

KantanBuildAnalyticsRegardless of what we do in our professional careers there is one thing that we all have in common, and that is how to get more done, be more productive and achieve the results we want…yesterday! For Machine Translation or Localization engineers this means finding the quickest way to get their MT engines ready to translate files.

KantanBuildAnalytics is a feature that solves the problem of how to quickly improve an engine after its initial training with minimum cost and effort. This post will teach you how to use KantanBuildAnalytics to get your KantanMT engines ready to translate faster.

Lets look at some of the features available for KantanBuildAnalytics:

  • Fluency Analysis – work with segment level BLEU scores to find out how relevant your training data is and how it impacts engine fluency.
  • Recall and Precision Analysis – use segment level F-Measure scores to understand the recall precision of your MT engines.
  • Post-Editing Estimation – calculate how much editing it will take to prepare a machine translated file for publishing using segment level TER (Translation Error Rate) scores.
  • Gap Analysis –  improve your engine quickly by creating terminology (glossary) files, simply download a list of untranslated words or ‘gaps’ (as an excel file) then re upload the excel files as new glossary training data.
  • Training Data Reject Reports – see any training data segments that have been rejected from the engine and their reason for rejection in a downloadable excel file.
  • Timeline – like your facebook timeline, see your MT engine’s history, with every action taken to improve the engine. It even lets you archive versions so if something goes wrong in the retraining, you can go back to an earlier version.

How to use KantanBuildAnalytics

Login into your KantanMT account using your email and your password.

You will be directed to the ‘My Client Profiles’ page. You will be in the ‘Client Profiles’ section of the ‘My Client Profiles’ page. The last profile you were working on will be ‘Active’.

My Client Profiles KantanMT
My Client Profiles Dashboard, KantanMT.com

To use ‘KantanBuildAnalytics’ with another profile other than the ‘Active’ profile. Click on the profile you want to use the ‘KantanBuildAnalytics’ with and make sure that the profile selected has at least one ‘Build’ job done successfully.

Then click on the ‘Build Analytics’ tab on the My Client Profiles’ page.

KantanBuildAnalytics
Selecting KantanBuildAnalytics™ on an active KantanMT profile.

This will take you to the ‘KantanBuildAnalytics’ page, where you will see the ‘Summary’ tab. This is selected by default. Your summary tab should give you an overview of the performance and measurement of your KantanMT engine.

And of course for the excel lovers, its possible to download the full summary report as an excel spreadsheet, so the engine’s performance information can be analysed to suit your organisation’s specific style requirements. To download the report click on the ‘Download summary report’ button.

To ‘Deep Tune’ the engine click on the ‘Deep Tune’ button. be warned though, this is a thorough tuning of the engine and will take a lot of time, the bigger the MT engine, the longer the tuning process takes.

KantanBuildAnalytics Summary Report
Download KantanBuildAnalytics Summary Report

A ‘Tune Engine’ pop up window will now appear on your screen, click on the ‘OK’ button if you want to deep tune or on ‘Cancel’ if you no longer wish to deep tune the engine.

To see how many segments in the training data were rejected, click on the ‘Rejects Report’ tab. This takes you to the ‘Rejects Report’ page, where you will see a list of segments and the reasons they were rejected.

KantanBuildAnalytics Rejects Report
Generating your KantanBuildAnalytics Rejects Report

To download an excel version of the rejects report click on the ‘Download’ button.

To create, test and manage customised preprocessing rules for your training data, click on the ‘Preprocessor Mngt’ button.

These features help MT or Localization Engineers build and develop better performing KantanMT engines. Read more about these features below, or Contact a member of our sales team,  to start using our platform now!

Essential KPIs for SMT: F-Measure

In our last blog post I discussed some of the Key Performance Indicators (KPIs) used by SMT developers to estimate the performance quality of their KantanMT engines. These KPIs help developers understand what aspects of their SMT engine are performing well and which need improvement.

In this blog I’m going to dive deep into F-Measure, a KPI which can provide insight into; the relevancy of your training data, the engine’s overall performance, and the suitability of an SMT engine for a particular domain or content type.

What is F-Measure?

F-Measure is a KPI which measures the precision and recall capabilities of an SMT system. It can also be viewed as a measure of translation accuracy and relevancy.

f-measure analagyBursting Red Balloons

In SMT, we can look at precision as a percentage of retrieved words that are relevant and recall (sometimes referred to as sensitivity) as the percentage of relevant words that are retrieved.

This is best explained using a thought experiment: So, imagine a box containing 10 red balloons and a few green balloons. Suppose we burst 5 balloons at random and 3 of these are red – we can calculate our precision as 3/5 (60%) and our recall as 3/10 (40%).

These two calculations offer a good estimation of the accuracy with which we are able to burst red balloons – the higher this calculation is, the better the chances that we will burst more red balloons.

So what has this thought experiment got to do with SMT systems?

Precision & Recall

Precision and recall are closely related to the understanding of accuracy.  Since SMT systems are based on pattern recognition, it is helpful to see how accurate they are at retrieving words and more importantly how relevant this retrieval is.

F-Measure is a calculation of both precision and recall and is expressed as a ratio.
If we go back to our balloon bursting experiment, precision was calculated as 60% and recall as 40%. To express these two values as a ratio, we can use the F-Measure formula as follows:-

f-measure     0.48

Source: Statistical Machine Translation by Philipp Koehn

In simple terms – we’re just not good at bursting red balloons 🙂

F-Measure and SMT engines

Using F-Measure we can get a general sense of the accuracy in which an SMT engine can retrieve words. If we examine the distribution of these scores across a set of reference translations we can get helpful insights which we can use to improve the training data and boost engine performance.

Here’s an example of an F-Measure distribution:

Statistical Machine Translation graph

Screen shot of Kantan BuildAnalytics F-Measure distributions

The overall F-Measure score for this particular SMT engine is 72%. This is a good value, and we can say that this engine is highly accurate at retrieving words for its target language and domain i.e. it has high precision in word retrieval and these are relevant to the target domain.

Also, the distribution of these scores across the reference translation set shows that the majority of these (60% of the total reference translations set) are in the 70-100% range. The distribution graph also shows that approximately 20% of the reference translations score less than 40%.  By examining this we can check to see if words/terminology are missing, and then create additional training material to improve the performance the engine.

Closing remarks…

F-Measure is a good starting point for understanding the quality of an SMT engine but it does have a major downfall, while it measures the recall and precision capabilities of an SMT engine, it doesn’t take into the account the order in which the words are retrieved.

So, as in the famous sketch with Andre Previn and Morecambe and Wise, we may know all the notes but not necessarily in the right order:

Morecambe_and_Wise_YT_screenshot

One more thing…
In order to improve the F-Measure score, an engine must become aware of word order, which is sometimes referred to as fluency. In the next post I will look at BLEU (Bilingual Evaluation Understudy) and examine how this metric helps us to further understand the quality of SMT engines.

KantanMT’s new BuildAnalytics technology illustrates the distribution of F-Measure, BLEU, and TER score across our members SMT engines. It also generates a Gap Analysis, highlighting missing words in members training data, and gives a provides KantanMT members with a training data rejects reports – great information that helps members of KantanMT.com develop a deep understanding of how their SMT engines work, and how to improve their performance.

You can watch a video of Kantan BuildAnalytics here>>