“If you talk to a man in a language he understands, that goes to his head. If you talk to him in his language, that goes to his heart.” Nelson Mandela

With those two simple sentences one of the 20^th century’s greatest icons summed up the power of language. Mandela understood that in speaking a person’s indigenous language you not only communicated from the brain but also from the heart. Mandela believed that is important for the world to rescue, maintain and help nurture the vast array of indigenous languages of this great planet of ours.

The General Assembly of the United Nations with UNESCO declared 2019 as the International Year of Indigenous Languages (IYIL). The purpose of the project is to promote awareness of the critical risks indigenous languages face, and to underline and promote their continued importance as “vehicles of culture, knowledge systems and ways of life”. These languages, activists would contend, are not mere repositories of words, grammar and lexicons but are indeed the foundations and cultural roots of many diverse peoples.

These indigenous languages play a critical role in empowering their communities. The use of their native tongue gives them a level playing field. It can, for instance, facilitate equitable democratic participation in their countries’ economic, cultural and political life. It is the means of making them inclusive of the body politic and drawing them into the heart their country’s system. It gives them hope and a stake in the future of their society.

Indigenous languages make up the majority of the world’s estimated 7,000 languages and, like all languages, they are as the UN argues “depositories and vehicles of culture, knowledge, values and identity”. The UN also maintains that the loss of any of these languages would be a deprivation for humanity, and a disempowerment for the communities specifically dependent on these languages. According to the New York Times of the estimated 7,000 languages spoken in the world today, linguists say, nearly half are in danger of extinction and are likely to disappear in this century. In fact, they are now falling out of use at a rate of about one every two weeks,

According to UNESCO, a language is endangered when parents are no longer teaching it to their children, and it is no longer being used in everyday life. (Source: “Dying languages: scientists fret as one disappears every 14 days“, https://www.thestar.com) A language is considered nearly extinct when it is spoken by only a few elderly native speakers. Languages with a mainly oral tradition have more chance of extinction than those with a written tradition.

What are the numbers behind indigenous languages?

• 7,000 languages spoken worldwide
• 370 million indigenous people in the world
• 90 countries with indigenous communities
• 5,000 different indigenous cultures
• 2680 languages in danger of extinction

According to UNESCO the Aims of the IYIL are:

1. Increasing understanding, reconciliation and international cooperation.
2. Creation of favourable conditions for knowledge-sharing and dissemination of good practices with regards to indigenous languages.
3. Integration of indigenous languages into standard setting.
4. Empowerment through capacity building.
5. Growth and development through elaboration of new knowledge.

It is through language that we communicate with the world. Language helps us define our identity, articulate our history and culture, and store our traditions. Indigenous language empowers people to defend their human rights and gives them the wherewithal to participate in an equitable way with all aspects of their society. In doing so, they give people a stake in, and a sense of ownership of, their society.

Through language, people preserve their community’s history, customs and traditions, folk memory, distinctive ways of thinking, meaning and expression.  Language is also used to construct their shared future. Language is pivotal in the areas of human rights protection, good governance, peace building, reconciliation, and sustainable development. (Source: https://en.iyil2019.org/role-of-language/)

A person’s right to use his or her chosen language is a prerequisite for freedom of thought, opinion and expression, access to education and information, employment, building inclusive societies, and other values enshrined in the Universal Declaration of Human Rights. Many of us take it for granted that we can conduct our lives in our native languages without any constraints or prejudice.  But this is not the case for everyone.

Of the almost 7,000 existing languages, the majority have been created and are spoken by indigenous peoples who represent the greater part of the world’s cultural diversity. Yet many of these languages are disappearing at an alarming rate, as the communities speaking them are confronted with cultural prejudice, linguistic assimilation, enforced relocation, educational disadvantage, poverty, illiteracy, migration and other forms of discrimination and human rights violations.

Given the complex systems of knowledge and culture developed and accumulated by these local languages over thousands of year, their disappearance would amount to losing a trove of cultural treasure.  It would deprive us of the rich diversity they add to our world and the ecological, economic and sociocultural contribution they make. More importantly, their loss would have a huge negative impact on the indigenous cultures concerned.

It is for this reason and others that the United Nations chose to dedicate a whole year to indigenous languages, to encourage urgent action to preserve, revitalise and promote them. Let’s hope it is a successful year.

This blog is based on information supplied by the project’s official site:

https://en.iyil2019.org/about#about-1

Where information on related events and what you can do to help can be found.

Aidan Collins is Marketing Manager at KantanMT.

One of the wonders of the modern I.T. era is the extent to which technology has shrunken this world. I.T. has given industry a global reach that can be measured in mere nanoseconds, or less. It is a brave new world. A reality that could only have been the stuff of science fiction only a few short decades ago. For example, when I started in this industry in 1990, we neither had the internet nor email, and I am no fossil!

In 1990 I started working with a localisation company who housed its own team of translations for the FIGS languages (French, Italian, German and Spanish – for the uninitiated). Those were pretty much the main languages requested by USA and European companies for localised products. That was the focus of their market at the time. After all, it is the market that drives the demand for products in local languages. Other localisation companies used the telephone and the fax, as well as couriers to manage their workflow. It was a clunky system by comparison to today, but it worked. What suffered was speed of delivery.

The revolutionary change in workflow practices was brought about by the introduction of the internet and email. Overnight, the localisation workflow model changed. No longer was it necessary to house teams of translators. In fact, it made economic sense not to. People eat up space – space eats up money for overheads – spending money puts up costs etc. And add to that dynamic new world of digital communication streams the Cloud, the Edge, and fibre optic cable connected to super powerful computers and the world suddenly becomes very tiny.

Today, instead of sitting in an office in Dublin, London or Amsterdam a translator can be virtually anywhere and still be available for work. Ironically, translation has reverted to something of a 19^th century kitchen industry work model. Most translators work from home. If they want to, they can sit on a beach and work. The important thing is they have connectivity and they deliver a quality product on time at the right price. With modern lightweight laptops (boy, where the old ones heavy!) and the seemingly ubiquitous WIFI no translator is restricted to where and when the translate.

And for those established translators it can be a lucrative business. The demand for them is high. Yet one of the banes of being a translator is the dreaded “downtime”, that period where they are waiting for a large project to arrive. While they wait it is difficult for them to take on other work unless it is small. The worst thing a translator can do is agree to do a large project and then announce at the last minute that they are no longer available. It destroys their reputation. So rather than put themselves in that position, most translators will suffer the downtime, a time when no money is being made.

Well, the good news is a new product from KantanMT.com may well make such downtime a thing of the past. The new product, which launches in October, is called KantanSkynet. The product is a “Human-Powered Machine Translation Platform”. It is a global crowd-sourcing translation platform, where anyone with a language skill can earn money. KantanSkynet lets team members improve machine translated texts in their spare time, also improving their own language skills, while working to a schedule that fits best with their lifestyle.

It is the aim KantanSkynet to provide human-powered translations to enhance MT-produced texts. Using advanced Artificial Intelligence (AI) technologies, KantanSkynet evaluates the quality of machine translated texts and routes low-rated quality texts to human translators for them to improve. These improvements are then used to fine-tune future machine translated (MT) texts, allowing the MT engines to continuously learn and so improve future work.

If you speak a foreign language, KantanSkynet will allow you to earn money by being part of this language improvement team. All you need is an internet-connected mobile device, and the KantanSkynet opportunity is ready for you to access. It’s a simple equation – the more you work, the better the language quality you provide, the more you’ll earn.

KantanSkynet also allows translators to work to their own schedule. They can work when they like, where they like and as often as they like. In doing so, they’ll improve our machine-translated texts and put their spare time to good use earning money. It is mutually beneficial.

Under this revolutionary workflow model, downtime will become a thing of the past for translators. For example, if a translator is travelling by train, they can logon and work for the duration of the journey. If they are between large projects, they can fill their time logging on and working on a KantanSkynet project. The bottom line is translators can now earn money anytime, anywhere simply by logging on and using their mobile device to improve machine-translated texts. It’s this simple: logon to a KantanSkynet-allocated account and start earning money anytime, anywhere! If you are a translator waiting for the next big project to land in your inbox don’t waste that time, use KantanSkynet to earn money in the interim.

And KantanMT will make sure that you get paid fast! – KantanSkynet will pay you as you work! There are no delays and you’re paid directly for work completed. The more you work, the higher the quality of your translation, the more you’ll get paid! It is your opportunity to be a part of the KantanSkynet translation community, providing the power and flexibility of machine translation coupled with the quality and authenticity of native language speakers like you. Get yourself an accountant and get earning.

Aidan Collins is Marketing Manager at KantanMT.

It started with the flu: not just any old strain of flu, but a virulent virus known by the name of H1N1. The year was 2009 and a public health epidemic was rocking the USA. And the authorities were sure if things were not controlled it could well become a pandemic (i.e. a country- or world-wide epidemic). The problem was the authorities were chasing the spread of the disease. By the time they had identified an area where it might appear it was already too late. The challenge then was how do you get ahead of the spreading disease, how do you treat it like a forest fire and surround it by a fire line that would stop it spreading? It seemed an impossible task. Step in Big Data and Google. No company in the world could do what Google could do in 2009. Google handled 3 billion search terms every day and had the huge processing power to do it. (Source: Big Data, Viktor Mayer-Schonberger & Kenneth Cukier).

Then someone had the brilliant idea of harnessing this intelligent to try and identify the path the spread of the epidemic might be taking, and as close to real-time as was possible under the circumstances. The idea was to identify search terms that people would use if they were worried about getting a flu or were beginning to feel the symptoms of the flu. The search words could be identified and the computational might of Google would allow them to identify 45 search terms that would set off an alarm, but more importantly it would allow them to identify the area where the people where located who were doing the searching. Suddenly, instead of chasing a spreading flu, the health authorities were able to identify probable hot spots and quickly deploy health professionals to treat the population ahead of any arriving flu bug. They were at last able to throw down a fire line and contain the spread. The pandemic would not happen – this time. This was the first time Big Data was harnessed in such a fashion. A new industry was born, and it has continued its relentless growth right up the present day. And it’s not about to disappear anytime soon.

So, what in a few words is Big Data? Well, the Big refers to the volume. Digital data has existed since the first bytes were input into a computer. Many computers had very large volumes of data on them. Think of all of the photos you have on hard drives, all of the documents you have created, all of the emails you have written, all of the Tweets you have tweeted, all of the Facebook messages you have posted and you are beginning to get an idea of the personal data you have created and are continuing to create, daily. Not multiply that the amount of people in the world who are doing the same thing. Here’s a figure for the active users of Facebook per day – 1.56 billion. (Source: Facebook DAU, March 2019). Add to that Twitter, Instagram, LinkedIn etc and you see where we are going with the numbers. Colossal! For me the numbers are incomprehensible, but I will share them with you: are you strapped in? Well, according to IBM there exists today 2.7 zettabytes of data online. A zettabyte is 1 billion terabytes – are you any clearer? To say that is a helluva lot of information would be something of an understatement.

That is one humongous potential source of business intelligence which could render an enormous variety of information and at a comparatively rapid speed. Using the right equipment, methodology and specially trained data scientists that information mother lode could be sliced, diced and parsed to uncover very valuable information. And every global company worth their salt wants to do just that. And the reason for that? – it is estimated that today’s online spending is $50 trillion per annum. Any company wanting even a fraction of that needs to have the ability to identify markets, trends, sentiments, opportunities and to do it damn quick. Have your grey cells exploded yet? Well, here’s one last figure for you – it is estimated that the volume of online business data doubles every 1.2 years! (Source: Users Bring Real Value to Big Data Machine Translation, Wired).

Yet what made all this possible. Why now in the 21^st century did it become a phenomenon? Well, the answer to that is the incredible explosion of the computational of computers. The development of fibre optic cable which allowed data to travel at fantastic speeds and the birth of the Cloud – a seemingly unlimited storage somewhere out there. Tie these elements to get and you get a completely different computation paradigm that hitherto existed. In addition to these technological rockets you had a change in mindset as to how Big Data could be used. It was as though a treasure chest (or Pandora’s box?) had been opened and made available to those who had the savvy as to how to exploit this global gift. The key word is global, as that is the key challenge. If a company truly wants to grab a slice of that $50 trillion-dollar market it needs to be able use online data, interrogate its own data collection, all of which exists in a multiple of languages. That is the reality of a global market. Only one third of those online are English speakers. It is estimated the other two thirds are covered by 45 other languages.

Enter Machine Translation – a technology that has existed since the 1960s and that has seen huge growth and refinement in the last 10 years. If a company wants to be truly global it must develop a way of handling huge volumes of data in multilingual formats and at fantastic speed – some even need it real time. The shear volume of what needs to be translated – to some degree or other, and not always perfectly – is, as we have seen – phenomenal. Machine Translation is the tool of choice for doing that. Not every translation needs to 100% perfect – companies like law firms need only “gist” translations. Other companies employing online chatlines need similar services – a translation with a quality that gets the message across. However, there are those who need perfection (if that exists – see my blog above on this). That too is being supplied more and more by MT AND Human Translators (HT). The latter will never disappear as a vital part of the translation equation. MT and HT are an essential partnership.

MT algorithms can crunch huge volumes of data and tremendous speeds and to an increasingly high-quality level. But we still need HT to bring the product to the accepted quality level (that can vary from customer to customer). The industry is giving shape to this new partnership translation paradigm. HT will always be a vital part of the translation/localisation industry. MT does not threaten their hard-earned status. I predict that the earning power of translators will rise over the coming short while. In theory, and I believe it will be in practice, translators will have the option to access work online 24/7/365. Only sleep will prevent them from accessing work when and where they want it. Things are developing rapidly behind the scenes and I believe that soon sky’s the limit for all translators out there.

Aidan Collins, Marketing Manager at KantanMT.

I am not a great fan of Formula 1 racing. For me it is a lot of going around in circles interspersed with some moments of confused drama. But the one part of it I do enjoy is the moment when all of those powerful machines are lined up on the starting grid. The highly tuned turbo-charged engines are roaring ready to spring forward at tremendous speeds when unleashed. It is wonderful to behold such a superb collection of state-of-the-art technology, brilliantly designed and programmed to achieve the pinnacle of success.

Well, according to the http://www.slator.com 2019 NMT industry report the domain of turbo-driven engines is not only to be found in Formula 1 racing. Apparently, developers of neural machine translation solutions are also “turbo-charging” their engines as they attempt to capture huge swathes of business in an ever-expanding market.

The report, published, earlier this year, titled “Neural Machine Translation Report: Deploying NMT in Operations”, says that there are now a dozen global tech companies “aggressively” pursuing enhanced solutions in machine translation and natural language processing. The list of global companies pouring millions into high-tech research reads like a Who’s Who of the IT world; Microsoft, Amazon, Salesforce, eBay and Facebook are just some of those who have poured millions into developing the technology.

For some – such as Microsoft and Amazon – the rationale behind their initial investment was self-serving, driven by internal needs to handle billions of words of their own translation projects. However, these global companies now see the advantages of taking this same self-built technology and monetizing it as a product they can now sell to smaller companies and also make available to individual users around the globe.

In parallel to this corporate push is a rapidly expanding level of research into the technology across many colleges in the USA, China and Europe. A chart in the report illustrates clearly this phenomenal growth, revealing that there was a notable rise in papers on NMT published in 2018. That year saw the publication of 391 papers, almost double that of 2017 and a six-fold increase of the 2016 numbers. Asia is to the forefront in the development of the technology.

Indeed, some Chinese academics controversially argue that the whole NMT industry was started by them and is now being propelled by their research. Undoubtedly, the exponential growth of the technology has been remarkable, growing from only a handful of champions just five years ago to a plethora of them today. The report talks of the current development spurt as being the “Third Wave” of MT technology. It suggests that with the huge investment by so many global companies – and growing levels of academic research – that this is a technology with a promising future.

As with the growth of any technology, there is the beginning of diversification processes as companies seek unique ways to monetize their product in an effort to recoup some of their investment. The industry is being driven by the global goliaths whose researchers were initially tasked with creating a solution for internal needs. These solutions are now seen as an opportunity for these corporations to develop another line of revenue. To harness this growing potential many of the tools’ companies, the companies behind such products as Catalyst, MemQ, Déjà Vu, Across, Wordfast etc have created APIs so that their product has the ability to interface smoothly with these NMT solutions.

Development efforts have also gone in many directions depending on the perceived needs of different markets. Some developers have concentrated energies in creating more and more language pairs; Google, for example, supports 50 pairs bidirectionally. Microsoft supports 41 pairs but uniquely allows its users to upload data if the do not have bilingual data. Amazon takes the lead by supplying up to 127 language pairs but is quite restrictive in how its users employ their engines.

One of the things all of these technology suppliers have it common is a menu of charges, although each menu differs as to when the charges kick in, and as to how is much is free. The report briefly highlights how many LSPs are struggling with how they should charge their customers for NMT services. Most have opted to go with the traditional per word method. However, the report suggests that this is something that might evolve into another method of charging as the service matures.

One thing that does jump out of the report is the number and scale of the Asian companies who are serious players in this field. In fact, Chief Scientist at Chinese search giant Baidu, Andrew Ng, claims that “Neural machine translation was a technology first pioneered and developed and shipped in China” and that US companies only came in “well after Baidu.” A comment that has stung many of the developers in the USA. The report seems to suggest there is a bit of a technology race between East and West development giants.

And as in the case of the huge sharks of the ocean, there are “pilot fish” companies swimming around the huge NMT companies seeking to feed off their efforts. This sub-industry is made up of companies who have seen an opportunity for supplying data-hungry developers with the billions of words they need from different market-types. Other enterprising companies have developed training courses aimed at addressing the growing demand for qualified post-editors. There has also been an increase in the number of “boutique” suppliers. These are smaller companies who offer the NMT technology to businesses who have neither the capital nor time to invest in developing their own NMT solution.

The report quotes several CEOs who argue that the technology will only become truly efficient when a comprehensive, qualitative form of integrated testing technology is available. The report interviews CEOs across a range of user companies and gives interesting feedback on their experiences in using NMT. Many of the reports are upbeat and give optimism that the technology is going in the right direction. A few of the CEOs have commented that after some initial reluctance many translators are now seeing that NMT is something that can help expand their earning power and are beginning to feel comfortable about where they now fit in the new L10n workflow.

The report is an easy read. It is not heavy on jargon, and it gives an interesting insight into the industry. Although the authors themselves don’t declare that machine translation technology is the future, taken in the round, it clear that NMT has become established, is making huge strides and is expanding from translating text to also translating voice. Clearly, this is an exciting time to be involved in the world of neural machine translation.

Aidan Collins, Marketing Manager at KantanMT.

When I started in L10n in 1990 the term L10n did not exist. That came later, after people got tired typing out the full word localisation (or localization). That’s the thing about L10n, time is always of the essence and ways to speed things up are always on the agenda. Over its history, L10n has always been driven by the imperative of dealing with projects quicker, cheaper and while still guaranteeing a high quality standard. In recent years, it is the challenge of dealing with huge volumes of data that has driven the latest technological revolution that is helping to reshape the L10n world. But more of that anon.

In earlier years projects consisted of three basic inter-related elements: documentation (user guides etc), translation (text from software, help-files and documents) and engineering (at that time more problematic and cumbersome, so often restricted in scope). In early years, engineers had nowhere near the plethora of tools they now have at their disposal. Some larger software companies developed proprietary internal tools to help software L10n, but these were restricted to internal staff and they did the heavy lifting on the engineering front.

Only when Catalyst came along (courtesy of the owner of KantanMT!) did engineering within L10n companies throw of its shackles (so to speak) and was able to blossom and grow as a discipline. Almost in parallel with that product we had the release of Trados. At first scorned by purist translators, in an almost Luddite-like fashion, it soon became an integral part of the L10n process. Slowly, translators saw the roles evolving to include the creation of glossaries and translation memories, and the management and coordination of these valuable translation memory assets.

In desk top publishing the PC market-share was pretty much dominated by Ventura, later bought from its original owners and shipped globally as Xerox Ventura. It was an effective workhorse and it dominated the market for the early part of the 1990s. Then came competitors like FrameMaker, which was Ventura on steroids, and then PageMaker and Quarkxpress for PC and the online publishing world took off, with a boost in its ability to handle and integrate complex graphics into documents and marry graphics, text and sound in to one product. The role of the desktop publisher went from someone churning out flat, grey pages to one where the skills of graphical engineering was required for that expanded role.

The history of localisation is marked with key technology milestones such as the introduction of Trados and Catalyst, and the expansion of documentation to become complex and online. Every milestone has also seen the evolution of the roles of the L10n practitioners, all the way from sales (selling new product offers), to engineering (will we build or deliver translated strings only), to DTP (will we go with the bells and whistles, or flat text?), to Project Management (how many more plates to the expect me to spin – cue workflow systems?), to finance (what to heck do we charge for translating strings? The same as text?), to the IT guys (more hardware please!), to the CEO (how do I finance and control this speed of expansion?). One truism is the L10n industry is always in a state of evolution and that is still the case.

Today, the technology driving things is machine translation (and the different flavours of it). MT is a technology that lay dormant because of the inability of companies to deal with its technical complexity and to fund the sophisticated machinery needed to make it mainstream. Then came a thing called the ‘Cloud’ (and now, its cousin the ‘Edge’) mixed with two other potent ingredients – powerful, economical hardware, and huge swathes of data for translation. Added to that already heady potion is the reality that global companies now want to speak to their customers in their own language, and almost at real-time, and you have an industry that has just been loaded with rocket fuel and is pointed on an upward trajectory.

Photo by Pixabay on Pexels.com

This technology milestone is also in the process of moulding and driving new roles, and interestingly – a new earning paradigm for translators. The latest MT drive is something that is causing translators in particular (and maybe quite a few CEOs) to fret about the viability of their profession. The good news is that any rumours of the pending extinction of human translators is just that – rumours without substance. I doubt in my lifetime (admittedly not as long as it used to be) will we see translation happening without supremely qualified translators somewhere in the mix. Indeed, and I make this prediction, translation and the role of translators is about to become more lucrative – if they choose to make it so.

And there won’t be long to wait for my prediction to become a reality. The days of translators having non-earning, down-time while they await the delivery (maybe next week, or maybe the week after or …) of the next major project, the one that will pay the mortgage, is over. I’ll go as far as to predict that translators will soon be in a position to earn money 24x7x365, should they choose. I believe because of MT and the Cloud their halcyon earning days await them.

And I can assure you this; my grey head is not in the clouds on this one.

Watch this space for further developments!

Aidan Collins, Marketing Manager at KantanMT.

We Saw in my previous blog that the first idea for Statistical Machine Translation was introduced by Warren Weaver as far back as 1947. As with many innovations, the idea was way ahead of the technological ability to make it happen. But with the growth of super powerful computers, the introduction of the Cloud as a repository for these super-computers, and the provision of these to LSPs at an affordable price permitted the theory of SMT as a solution from translation of languages to become commonplace.

But as with all technological breakthroughs, once the tipping point is reached things grow exponentially. While SMT could provide a translation model – albeit in a limited capacity – scientist began to look at the next iteration of machine translation – Neural Machine Translation.

A generic neural network is a real or virtual computer system that is designed to mirror the brain in its ability to “learn” to analyse bodies of complex data. Neural networks, particularly those with multiple hidden layers (hence the term ‘deep’ – the more layers, the deeper) are remarkably good at learning input->output mappings.

Such systems are composed of input nodes, hidden nodes, and output nodes that can be thought of as loosely analogous to biological neurons, albeit greatly simplified, and the connections between nodes can be thought of as in some way reflecting synapse connections between neurons in the human brain. As stated, the ‘deep’ element of the concept refers to a multi-layered processing network of neuron filters.

In the human process, instructions and information flows from the brain through neurons connected by synapses. In the machine translation equivalent artificial neurons are used to fine-tune and refine input data as it is passed through the translation ‘engine’ with a view to achieving a predicted output. In addition, the process called Deep Learning learns – as does the human brain – from experience and can adjust and remember processes accordingly. Consequently, the more an engine is used the more refined and powerful it becomes. The process becomes a life-long R&D development.

So, in translation, the language engine uses deep neural networks to essentially predict the next word in a sequence. The Neural Network is built using a bi-lingual corpus and is trained on full sentences. (These are referred to a sequences.) This is important, as NMT fully understands what a sentence is, whereas SMT only understands what a phrase is. This shift to training Neural Networks on sequences means that we eradicate almost all the syntactical and grammatical errors frequently found in SMT outputs.

For example, NMT systems make significantly fewer errors when compared to SMT systems:

(NOTE: every graph below 0 is the percentage decrease in error types for NMT systems compared to SMT systems.)

Therefore, NMT provides a demonstrable improvement in machine translation outputs, resolving many of the problems associated with the statistical machine translation (SMT) model. Effectively, in the last two years it has solved many of the translation shortcomings of the SMT system; deficiencies that SMT users have been trying to resolve for the last two decades. NMT-based R&D has been moving with a speed far out-pacing the slow pace of SMT development.

Amazingly, Google managed to condense in to four years the development of its NMT solution. That compares with 15 years it spent on developing and refining statistical machine translation (SMT). This quantum leap in output quality for NMT systems is the result of being able to train the translation models on ‘sequences’ (i.e. full sentences) rather than words, and the use of deep learning to train and refine the neural networks on an ongoing basis.

So, a Neural Network is a mathematical model seeking to imitate how the human brain functions. Obviously, scientists can only do this in a modest way. The human brain consists of approximately 100 billion neurons, and these are highly connected to other neurons creating super-complex relationships. In a Neural Network a neuron is a ‘word’ within a sequence (sentence), and each word is highly connected to words within the same sequence creating a complex web of grammatical and syntactical relationships.

By creating these neutral models with lots of sequences, we can build up an elaborate network of word relationships. We can use this network to predict translations within the context of the training data used to build the translation model.

To determine if a model is good at predicting translations (relative to the training data), we use a mathematical number called Perplexity. We want to build networks with a Perplexity as low as possible. If we achieve this, we can predict good quality translations that contain fewer syntactical and grammatical errors when compared to SMT systems.

These highly efficient models can then provide fast and fluent translations, and at economically affordable prices. Today, over 90% of the daily traffic on the KantanMT platform is processed by our NMT services. Slator.com has reported that the number of LSPs now using NMT has quadrupled since 2018. This gives an indication of the high regard customers have for the efficacy of the NMT platform. The powers of deep learning, the availability of huge amounts of data and the ever-increasing processing power of computers makes it inevitable that this branch of artificial intelligence will be with us for the long haul.

One of the concomitant effects of the introduction and widespread use of viable machine translation solutions within LSPs is the inevitable re-organising of the translation process, and the realignment of existing jobs to fit with the newest technology trends. I will look at this in my next blog and try to predict how job roles might evolve over the coming years.

Aidan Collins, Marketing Manager at KantanMT.

The first ideas of Statistical Machine Translation were introduced by Warren Weaver as far back as 1947. He explained that language had an inherent logic that could be treated in the same way as any logical mathematical challenge. He contended that logical deduction could be used to identify “conclusions” in the target (untranslated) language based on what already existed in the source (translated) language. With the advent of the cloud, and affordability of powerful computers, the theory of Statistical Machine Translation (SMT) became a practical option.

Therefore, SMT is a machine translation paradigm where translations are generated based on statistical models, whose parameters are derived from the analysis of bilingual text corpora (text bodies) – a source text of translated material and a target text of untranslated material.

Statistical machine translation starts with a very large data set of approved previous translations. This is known as a corpus (corpora is plural) of texts that is then used to automatically deduce a statistical model of translation. This model is then applied to untranslated target texts to make a probability-driven match to suggest a reasonable translation.

Where are these large data sets sourced? Well, over many years most large global organisations, for example the EU, UN, World Bank, World Health Organisation etc develop enormous domain-specific corpora, in multiple source/target language combinations. Many of these have originated as human-translated texts. These are then made accessible to machine translation users/developers to further refine and use for MT purposes. The process is an evolutionary model with each existing corpus being refined, added to and updated on an ongoing basis.

This SMT paradigm is based on what’s known as ‘probabilistic mathematical theory’. Such theory suggests the chances (probability) of something occurring depending of different variables likely to influence the event. For example, tossing a dice gives the probability of one dice being a certain number is 1/6. So, a gambler has a one in six chance of landing on their chosen number. The probability of two dice being the same number is 1/6 x 1/6 = 1/36. In poker, the professional gambler keeps mental track of the cards being played and using probability theory he/she decides whether a gamble has a good chance of winning based on the odds they have calculated in their head.

In SMT, the MT engineer builds a Translation Model using the frequency of phrases appearing the training corpus into a table. This table stores the phrase and the number of times this repeats over the entirety of the training corpus. The more frequently a phrase is repeated in a training corpus, the more probable the target translation is correct. Each phrase (stored in the Phrase Table) can range from one to five words in length. This phrase table is referred to as the Translation Model.

Consequently, the MT engineer is using a probability model to hit on the right source/target translation combination. The process is evolutionary as the corpus is refined and adjusted after each translation run to eliminate/adjust any anomalies. The more frequently the corpus is used, the more perfected it becomes. The development of the corpus quality is a continuous organic R&D process of a highly valuable translation asset.

Additionally, the MT engineer builds a secondary model using the target translation data. This model helps determine the order in which the engineer needs to assemble phrases (from the Phrase Table) in order to optimise translation Fluency i.e. to give the translated text its natural language flow. Fluency ensures that literal translations (i.e. the words are all there, but the sense of the sentence is not) are replaced by a more natural sounding translation.

In order to translate a source sentence, the MT engineer goes through the following decode process (as outlined in the diagram below):

1. He/she breaks the source language sentence into phrases. (You can see the phrase as individual grey blocks in line two of the diagram.)
2. He/she then looks up each of these phrases in the Phrase Table/Translation Model and generates the target language translations. (You can see this in line three of the diagram.)
3. The engineer then uses the Phrase Table/Translation Model to re-order these phrases to optimise translation Fluency. (You can see this reordering in line four of the diagram.)

In my next blog, I will give a short introduction to the increasingly popular MT model Neural Machine Translation.

Aidan Collins, Marketing Manager at KantanMT.