5 Best Practices for Checking Machine Translation Quality

Excel KantanMT

You have your finger on the pulse of latest technologies, and you are proud to use the latest automated technology for your localization needs. But, sometimes it might feel like you are still stuck in the 90s when it comes to reviewing your Machine Translation (MT) output for quality – especially, if you are using spreadsheets to collate your reviewers’ feedback on segments.

Traditionally language quality review for MT involves the Project Managers (PMs) sending copies of a static spreadsheet to a team of translators. This spreadsheet contains lines of source and target segments, with additional columns where the reviewers score the translated segments according to a set of predefined parameters.

Once the spreadsheets are sent off to the reviewers, PMs are completely in the dark – with no idea how the reviewers are progressing, when they might complete the review, or if they have even started the project.

If that sounds tiring, imagine what the PM has to go through!

Continue reading

Create, Test and Deploy Post-Editing Automation Rules with KantanMT PEX Rule Editor

The KantanPEX Rule Editor enables members of KantanMT reduce the amount of manual post-editing required for a particular translation by creating, testing and deploying post-editing automation rules on their Machine Translation engines (client profiles).

The editor allows users to evaluate the output of a PEX (Post-Editing Automation) rule on a sample of translated content without needing to upload it to a client profile and run translation jobs. Users can enter up to three pairs of search and replace rules, which will be run in descending order on your content.

How to use the KantanMT PEX Rule Editor

Login into your KantanMT account using your email and your password.

You will be directed to the ‘Client Profiles’ tab in the ‘My Client Profiles’ page.  The last profile you were working on will be ‘Active’ and marked in bold.

Active Profile, KantanMT, Client Profile

To use the ‘PEX-Rule Editor’ with a profile other than the ‘Active’ profile, click on the  new profile name to select that profile for use with the ‘Kantan PEX-Rule editor’.

Then click the ‘KantanMT’ tab and select ‘PEX Editor’ from the drop-down menu.

Client Profile, KantanMT, PEX Editor

You will be directed to the ‘PEX Editor’ page.

Type the content you wish to test on, in the ‘Test Content’ box.

Test Content, PEX Rule Editor, KantanMT

Type the content you wish to search for in the ‘PEX Search Rules’ box.

PEX Search Rules, KantanMT, PEX Editor

Type what you want the replacement to be in the ‘PEX Replacement Rules’ box and click on the ‘Test PEX Rules’ button to test the PEX-Rules.

PEX Replacement Rules, Pex Editor , KantanMt , Products

The results of your PEX-Rules will now appear in the ‘Output’ box.

Output Content , PEX Rule Editor

Give the rules you have created a name by typing in the ‘Rule Name’ box.

Rule Name, PEX Rule Editor , KantanMT

Select the profile you wish to apply this rule(s) to and then click on the ‘Upload Rule’ button.

Profile and Button, KantanMT , PEX

Additional Information

KantanMT PEX editor helps reduce the amount of manual post-editing required for a particular translation, hence, reducing project turn-around times and costs. For additional information on PEX-RULES and the Kantan PEX-Rule editor please click on the links below. For more details about  KantanMT localization products  and ways of improving work productivity and efficiency please contact us at info@kantanmt.com.

 

Using F-Measure in Kantan BuildAnalytics

What is F-Measure ?

KantanMT Logo 800x800 F-Measure is an automated measurement that determines the precision and recall  capabilities of a KantanMT engine. F-Measure measures enables you to determine the  quality and performance of your KantanMT engine

  • To see the accuracy and performance of your engine click on the ‘F-measure Scores’ tab. You will now be directed to the ‘F-measure Scores’ page.

F-Measure tab

  • Place your cursor on the ‘F-measure Scores Chart’ to see the individual score of each segment. A pop-up will now appear on your screen with details of the segment under these headings, ‘Segment no.’, ‘Score’, ‘Source’, ‘Reference/Target’ and ‘KantanMT Output’.

Segment

  • To see the ‘F-measure Scores’ of each segment in a table format scroll down. You will now see a table with the headings ‘No’, ‘Source’, ‘Reference/Target’, ‘KantanMT Output’ and ‘Score’.
  • To see an even more in depth breakdown of a particular ‘Segment’ click on the Triangle beside the number of the segment you wish to view.Triangle
  • To reuse the engine as Test Data click on the ‘Reuse as Test Data’. When you do so, the ‘Reuse as Test Data’ button will change to ‘Delete Test Data’.Test Data
    Delete Test Data
  • To download the ‘F-measure Scores’, ‘BLEU Score’ and ‘TER Scores’ of all segments click on the ‘Download’ button on either the ‘F-measure Scores’, ‘BLEU Score’ or ‘TER Scores’ page.download

This is one of the features provided by Kantan BuildAnalytics to improve an engine’s quality after its initial training .To see other features used by Kantan BuildAnalytics please click on the link below .To get more information about KantanMT and the services we provide please contact our support team at  at info@kantanmt.com.

What is KantanISR and Why do I need it ?

KantanISR technology enables KantanMT members to perform instant segment retraining using a pop-up editor. The technology is designed to permit the near-instantaneous submission of post-edited translations into a KantanMT engine so that KantanMT members can submit segments for retraining, hence bypassing the need to completely rebuild the engine.

KantanISR was developed with usability, efficiency and productivity in mind as members simply need to login to their KantanMT account, go to their main dashboard and submit new training segments using the KantanISR Editor. This adding of high quality training data to a KantanMT engine will improve the translation quality of that engine and reduce post-editing requirements.

Using KantanISR

      1. Login into your KantanMT account using your email and your password.
      2. You will be directed to the ‘My Client Profiles’ page. You will be in the ‘Client Profiles’section of the ‘My Client Profiles’ page. The last profile you were working on will be‘Active’.
      3. If you wish to use the ‘KantanISR’ with another profile other than the ‘Active’ profile. Click on the profile you wish to use the ‘KantanISR’ with, then click on the ‘Training Data’ tab.
      4. You will be directed to the ‘Training Data’ page. Now click on the ‘IRS’ tab.
      5. The ‘KantanISR’ wizard will now pop-up on your screen.
      6. Add the source language text in the ‘Source’ text editor fields. Add the corresponding target language text in the ‘Target’ text editor fields.
      7. Then click on the ‘Save’ button if your happy with your retraining data. If not click the‘Cancel’ button.
      8. When you click the save button a ‘KantanISR successful’ pop-up will appear on your screen, click the ‘OK’ button and you will be directed back to the ‘Training Data’ page.

Using KantanISR through KantanAPI

Please Note: The KantanAPI is only available to KantanMT members in the Enterprise Plan.

Members’ can also get the benefit of KantanISR through KantanAPI by using HTTP

GET requests. The API expects:

  • A user authorisation token (‘API token’) which can be gotten by clicking on the ‘API’
  • The name of the client profile you wish to use.
  • A source segment and its target segment in the languages specified when profile was created.

To learn more about KantanISR or get help with KantanMT technologies, please contact us at info@kantanmt.com. Hear from the Development team on why KantanISR increases productivity and efficiency for KantanMT customers.

 

What is KantanBuildAnalytics™?

KantanBuildAnalyticsRegardless of what we do in our professional careers there is one thing that we all have in common, and that is how to get more done, be more productive and achieve the results we want…yesterday! For Machine Translation or Localization engineers this means finding the quickest way to get their MT engines ready to translate files.

KantanBuildAnalytics is a feature that solves the problem of how to quickly improve an engine after its initial training with minimum cost and effort. This post will teach you how to use KantanBuildAnalytics to get your KantanMT engines ready to translate faster.

Lets look at some of the features available for KantanBuildAnalytics:

  • Fluency Analysis – work with segment level BLEU scores to find out how relevant your training data is and how it impacts engine fluency.
  • Recall and Precision Analysis – use segment level F-Measure scores to understand the recall precision of your MT engines.
  • Post-Editing Estimation – calculate how much editing it will take to prepare a machine translated file for publishing using segment level TER (Translation Error Rate) scores.
  • Gap Analysis –  improve your engine quickly by creating terminology (glossary) files, simply download a list of untranslated words or ‘gaps’ (as an excel file) then re upload the excel files as new glossary training data.
  • Training Data Reject Reports – see any training data segments that have been rejected from the engine and their reason for rejection in a downloadable excel file.
  • Timeline – like your facebook timeline, see your MT engine’s history, with every action taken to improve the engine. It even lets you archive versions so if something goes wrong in the retraining, you can go back to an earlier version.

How to use KantanBuildAnalytics

Login into your KantanMT account using your email and your password.

You will be directed to the ‘My Client Profiles’ page. You will be in the ‘Client Profiles’ section of the ‘My Client Profiles’ page. The last profile you were working on will be ‘Active’.

My Client Profiles KantanMT
My Client Profiles Dashboard, KantanMT.com

To use ‘KantanBuildAnalytics’ with another profile other than the ‘Active’ profile. Click on the profile you want to use the ‘KantanBuildAnalytics’ with and make sure that the profile selected has at least one ‘Build’ job done successfully.

Then click on the ‘Build Analytics’ tab on the My Client Profiles’ page.

KantanBuildAnalytics
Selecting KantanBuildAnalytics™ on an active KantanMT profile.

This will take you to the ‘KantanBuildAnalytics’ page, where you will see the ‘Summary’ tab. This is selected by default. Your summary tab should give you an overview of the performance and measurement of your KantanMT engine.

And of course for the excel lovers, its possible to download the full summary report as an excel spreadsheet, so the engine’s performance information can be analysed to suit your organisation’s specific style requirements. To download the report click on the ‘Download summary report’ button.

To ‘Deep Tune’ the engine click on the ‘Deep Tune’ button. be warned though, this is a thorough tuning of the engine and will take a lot of time, the bigger the MT engine, the longer the tuning process takes.

KantanBuildAnalytics Summary Report
Download KantanBuildAnalytics Summary Report

A ‘Tune Engine’ pop up window will now appear on your screen, click on the ‘OK’ button if you want to deep tune or on ‘Cancel’ if you no longer wish to deep tune the engine.

To see how many segments in the training data were rejected, click on the ‘Rejects Report’ tab. This takes you to the ‘Rejects Report’ page, where you will see a list of segments and the reasons they were rejected.

KantanBuildAnalytics Rejects Report
Generating your KantanBuildAnalytics Rejects Report

To download an excel version of the rejects report click on the ‘Download’ button.

To create, test and manage customised preprocessing rules for your training data, click on the ‘Preprocessor Mngt’ button.

These features help MT or Localization Engineers build and develop better performing KantanMT engines. Read more about these features below, or Contact a member of our sales team,  to start using our platform now!

Translation Quality: How to Deal with It?

KantanMTSelcuk Ozcan, Transistent, KantanMT started the New Year on a high note with the addition of the Turkish Language Service Provider, Transistent to the KantanMT Preferred MT Supplier partner program.

Selçuk Özcan, Transistent’s Co-founder has given KantanMT permission to publish his blog post on Translation Quality. This post was originally published in Dragosfer and the Transistent Blog.

 

 

Literally, the word quality has several meanings, one of them being “a high level of value or excellence” according to Merriam-Webster’s dictionary. How should one deal with this idea of “excellence” when the issue at hand is translation quality? What is required, it looks like, is a more pragmatic and objective answer to the abovementioned question.

This brings us to the question “how could an approach be objective?” Certainly, the issue should be assessed through empirical findings. But how? We are basically in need of an assessment procedure with standardized metrics. Here, we encounter another issue; standardization of translation quality. From now on, we need to associate these concepts with the context itself in order to make them clear.

Image 1 blog Transistent

Monolingual issues

Bilingual issues

As it’s widely known, three sets of factors have an effect on the quality of the translation process in general. Basically, analyzing source text’s monolingual issues, target text’s monolingual issues and bilingual issues defines the quality of the work done. Nevertheless, the procedure should be based on the requirements of the domain, audience and linguistic structure of both languages (source and target); and in each step, this key question should be considered: ‘Does the TT serve to the intended purpose?’

We still have not dealt with the standardization and quality of acceptable TT’s. The concept of “acceptable translation” has always been discussed throughout the history of translation studies. No one is able to precisely explain the requirements. However, a further study on dynamic QA models needs to go into details.There are various QA approaches and models. For most of them, acceptable translation falls into somewhere between bad and good quality, depending on the domain and target audience. The quality level is measured through the translation error rates developed to assess MT outputs (BLEU, F-Measure and TER) and there are four commonly accepted quality levels; bad, acceptable, good and excellent.

The formula is so simple: the TT containing more errors is considered to be worse quality. However, the errors should be correlated with the context and many other factors, such as importance for the client, expectations of the audience and so on. These factors define the errors’ severity as minor, major, and critical. A robust QA model should be based upon accurate error categorization so that reliable results may be obtained.

We tried to briefly describe the concept of QA modeling. Now, let’s see what’s going on in practice. There are three publicly available QA models which inspired many software developers on their QA tool development processes. One of them is LISA (Localization Industry Standards Association) QA Model. The LISA Model is very well known in the localization and translation industry and many company-specific QA models have been derived from it.

The second one is J2450 standard that was generated by SAE (Society for Automotive Engineers) and the last one is EN15038 standard, approved by CEN (Comité Européen de Normalisation) in 2006. All of the above mentioned models are the static QA models. One should create his/her own frameworks in compliance with the demands of the projects. Nowadays, many of the institutes have been working on dynamic QA models (EU Commission and TAUS). These models enable creating different metrics for several translation/localization projects.

About Selçuk Özcan

Selçuk Özcan has more than 5 years’ experience in the language industry and is a co-founder of Transistent Language Automation Services. He holds degrees in Mechanical Engineering and Translation Studies and has a keen interest in linguistics, NLP, language automation procedures, agile management and technology integration. Selçuk is mainly responsible for building high quality production models including Quality Estimation and deploying the ‘train the trainers’ model. He also teaches Computer-aided Translation and Total Quality Management at the Istanbul Yeni Yuzyil University, Translation & Interpreting Department.

Read More about KantanMT’s Partnership with Transistent in the official News Release, or if you are interested in joining the KantanMT Partner Program, contact Louise (info@kantanmt.com) for more details on how to get involved. 

Transistent KantanMT Preferred MT Supplier

 

Essential KPIs For Your SMT Engines

key performance indicator SMT engineThere are several key metrics that developers of Statistical Machine Translation (SMT) engines need to pay attention to. These key performance indicators (KPIs) help you to understand which aspects of an engine are performing well and which need improvement. They also provide insight into how you can improve the overall performance of your SMT engine.

SMT KPIs

While the following list is not exhaustive, it is a good starting point in your understanding of SMT performance and how to improve it.  You should remember that no single KPI can be analysed individually when determining the quality of an engine – they need to be analysed together, in a holistically manner, to give an accurate sense of the overall performance of the SMT engine.

F-Measure

F-Measure is a KPI that is used to determine the recall and precision capabilities of an SMT engine. Put simply, this measures how many words are picked from the SMT engine and how accurate the selection process is. Expressed as a ratio, F-Measure provides a good insight into the language coverage of an SMT engine.

If your F-Measure score is low, it indicates your engine is missing many words – if the score is high, it means that most words have been found in your engine.

While most SMT vendors present a single F-Measure value for an SMT engine, more progressive suppliers of SMT systems provide a distribution analysis of F-Measure scores.

Unfortunately, the level of precision and recall doesn’t give us information about the word order of a segment, for this we need to look at another KPI commonly referred to as BLEU.

BLEU

BLEU (Bilingual Evaluation Understudy) is a KPI that measures the fluency of the translated output of an SMT engine, which means it measures how many words overlap in a given translation when compared to a reference translation. Higher scores are given to segments which contain a greater number of sequential words.

BLEU is a major improvement on F-Measure as it takes word-order in account!

A high BLEU KPI means that an SMT engine is producing highly fluent translations; a low score means that it’s generally producing garbage. BLEU scores of 60% or higher are normally required before any SMT engine is considered production ready.

BLEU Score is easy use and understand, it is language independent and correlates highly with human evaluation which is why it is the most widely used KPI determining the quality of SMT engines.

TER

TER stands for Translation Error Rate and is an important KPI used to predict the most likely post-editing effort required for an SMT engine. It basically counts the number of insertions, deletions and substitutions that are required to transform a translation into a reference translation.

This is essentially what a professional translator would do in order to post-edit a translation to a level of publishable quality. Since post-editing is a timely and costly activity, SMT developers will try to minimize this and aim for low TER scores; less than 40% is a good benchmark for this KPI.

One more thing…

While older SMT systems produce these KPIs as single numerical values, a more modern approach is to look at the distribution of these KPIs across an SMT engine. This provides deeper insights and more accurate analysis of how an SMT engine is most likely going to perform in a production environment.

Three Take-aways:

1. Always aim for a High BLEU score, a High F-Measure score and a Low TER score.

2. Look at all three scores together to get a more holistic assessment of your SMT engine.

3. Examine the distribution of these KPIs across your SMT engine – this will help you to make smarter data choices for future customizations.

In our next blog we shall take a detailed look at each of these scores and see how we can work with them to improve SMT systems.

Tony O’Dowd, Founder and Chief Architect, KantanMT