Normative language work in the age of machine learning

Abstract

Neural nets have, during the last few years, given us both an improved Google Translate, better search algorithms, better speech technology and doubtless many other things. The approach dominates current language technology to the extent that no other approach is visible. Being data driven, the hidden assumption behind this approach when used in proofing tools is that the language is used correctly in the text material, in other words, usage equals the norm. Although this approach is able to provide useful help for the largest languages, it leads to some serious problems. For indigenous and often also for other minority languages, the assumption does not hold. The written norm is weakly established and cannot be reliably found in usage. For normative bodies responsible for defining the written norm of a given language, usage-based proofing tools will not be able to implement the explicit norm they have defined. The present article discusses the current trend within proofing tools and looks at some alternatives.