ptr727.LanguageTags 1.0.44

LanguageTags

C# .NET library for ISO 639-2, ISO 639-3, RFC 5646 / BCP 47 language tags.

Build Status

Code and Pipeline is on GitHub

NuGet Package

Packages published on NuGet

Introduction

This project serves two primary purposes:

• Publishing ISO 639-2, ISO 639-3, RFC 5646 language tag records in JSON and C# format.
• Code for IETF BCP 47 language tag construction and parsing per the RFC 5646 semantic rules.

Terminology clarification:

• An IETF BCP 47 language tag is a standardized code that is used to identify human languages on the Internet.
• The tag structure is standardized by the Internet Engineering Task Force (IETF) in Best Current Practice (BCP) 47.
• RFC 5646 defines the BCP 47 language tag syntax and semantic rules.
• The subtags are maintained by Internet Assigned Numbers Authority (IANA) Language Subtag Registry.
• ISO 639 is a standard for classifying languages and language groups, and is maintained by the International Organization for Standardization (ISO).
• RFC 5646 incorporates ISO 639, ISO 15924, ISO 3166, and UN M.49 codes as the foundation for its language tags.

Note that the implemented language tag parsing and normalization logic may be incomplete or inaccurate.

Refer to Language Tag Libraries for other known implementations.
Refer to References for specification details.

Build Artifacts

The build tool downloads language tag data files, converts them into JSON files for easy consumption, and generates C# classes with all the tags for direct use in code.

• ISO 639-2: Source, Data, JSON, Code
• ISO 639-3: Source, Data, JSON, Code
• RFC 5646 : Source, Data, JSON, Code

The data files are updated weekly using a scheduled actions job.

Usage

Tag Format

Refer to RFC 5646 Section 2.1 for complete language tag syntax and rules.

IETF language tags are constructed from sub-tags in the form of:

• Normal tags:
  • [Language]-[Extended language]-[Script]-[Region]-[Variant]-[Extension]-[Private Use]
  • Language:
    • See RFC 5646 Section 2.2.1
    • 2 - 3 alpha: Shortest ISO 639 code
    • 4 alpha: Future use
    • 5 - 8 alpha: Registered tag
  • Extended language:
    • See RFC 5646 Section 2.2.2
    • 3 alpha: Reserved ISO 639 code
  • Script:
    • See RFC 5646 Section 2.2.3
    • 4 alpha: ISO 15924 code
  • Region:
    • See RFC 5646 Section 2.2.4
    • 2 alpha: ISO 3166-1 code
    • 3 digit: UN M.49 code
  • Variant:
    • See RFC 5646 Section 2.2.5
    • 5 - 8 alphanumeric: Registered tag
  • Extension: ([singleton]-[extension])
    • See RFC 5646 Section 2.2.6
    • 1 alphanumeric: Singleton
    • 2 - 8 alphanumeric: Extension
  • Private Use: (x-[private])
    • See RFC 5646 Section 2.2.7
    • x: Singleton
    • 1 - 8 alphanumeric: Private use
• Grandfathered tags:
  • See RFC 5646 Section 2.2.8
  • Grandfathered tags are converted to current form tags
  • E.g. en-gb-oed → en-GB-oxendict, i-klingon → tlh.
• Private use:
  • All tags are private use
  • x-[private]-[private]

Examples:

• zh : [Language]
• zh-yue : [Language]-[Extended language]
• zh-yue-hk: [Language]-[Extended language]-[Region]
• hy-latn-it-arevela: [Language]-[Script]-[Region]-[Variant]
• en-a-bbb-x-a-ccc : [Language]-[Extension]-[Private Use]
• en-latn-gb-boont-r-extended-sequence-x-private : [Language]-[Script]-[Region]-[Variant]-[Extension]-[Private Use]

Tag Lookup

Tag records can be constructed by calling Create(), or loaded from data LoadData(), or loaded from JSON LoadJson(). The records and record collections are immutable and can safely be reused and shared across threads.

Each class implements a Find(string languageTag, bool includeDescription) method that will search all tags in all records for a matching tag.
This is mostly a convenience function and specific use cases should use specific tags.

Iso6392Data iso6392 = Iso6392Data.Create();
Iso6392Data.Record record = iso6392.Find("afr", false)
// record.Part2B = afr
// record.RefName = Afrikaans
record = iso6392.Find("zulu", true)
// record.Part2B = zul
// record.RefName = Zulu

Iso6393Data iso6393 = Iso6393Data.LoadData("iso6393");
Iso6393Data.Record record = iso6393.Find("zh", false)
// record.Id = zho
// record.Part1 = zh
// record.RefName = Chinese
record = iso6392.Find("yue chinese", true)
// record.Id = yue
// record.RefName = Yue Chinese

Rfc5646 rfc5646 = Rfc5646.LoadJson("rfc5646.json");
Rfc5646.Record record = rfc5646.Find("de", false)
// record.SubTag = de
// record.Description = German
record = iso6392.Find("zh-cmn-Hant", false)
// record.Tag = zh-cmn-Hant
// record.Description = Mandarin Chinese (Traditional)
record = iso6392.Find("Inuktitut in Canadian", true)
// record.Tag = iu-Cans
// record.Description = Inuktitut in Canadian Aboriginal Syllabic script

Tag Conversion

Tags can be converted between ISO 639 and IETF forms using GetIetfFromIso() and GetIsoFromIetf().
Tag lookup will use the user defined Overrides map, or the tag record lists, or the local system CultureInfo.
If a match is not found the undetermined und tag will be returned.

LanguageLookup languageLookup = new();
languageLookup.GetIetfFromIso("afr"); // af
languageLookup.GetIetfFromIso("zho"); // zh

LanguageLookup languageLookup = new();
languageLookup.GetIsoFromIetf("af"); // afr
languageLookup.GetIsoFromIetf("zh-cmn-Hant"); // chi
languageLookup.GetIsoFromIetf("cmn-Hant"); // chi

Tag Matching

Tag matching can be used to select content based on preferred vs. available languages.
E.g. in HTTP Accept-Language and Content-Language, or Matroska media stream LanguageIETF Element.

IETF language tags are in the form of [Language]-[Extended language]-[Script]-[Region]-[Variant]-[Extension]-[Private Use], and sub-tag matching happens left to right until a match is found.

Examples:

• pt will match pt Portuguese, or pt-BR Brazilian Portuguese, or pt-PT European Portuguese.
• pt-BR will only match pt-BR Brazilian Portuguese\
• zh will match zh Chinese, or zh-Hans simplified Chinese, or zh-Hant for traditional Chinese, and other variants.
• zh-Hans will only match zh-Hans simplified Chinese.

LanguageLookup languageLookup = new();
languageLookup.IsMatch("en", "en-US"); // true
languageLookup.IsMatch("zh", "zh-cmn-Hant"); // true
languageLookup.IsMatch("sr-Latn", "sr-Latn-RS"); // true
languageLookup.IsMatch("zha", "zh-Hans"); // false
languageLookup.IsMatch("zh-Hant", "zh-Hans"); // false

Tag Builder

The LanguageTagBuilder class supports fluent builder style tag construction, and will return a constructed LanguageTag class through the final Build() or Normalize() methods.

The Build() method will construct the tag, but will not perform any correctness validation or normalization.
Use the Validate() method to test for shape correctness. See Tag Validation for details.

The Normalize() method will build the tag and perform validation and normalization. See Tag Normalization for details.

LanguageTag languageTag = new LanguageTagBuilder()
    .PrimaryLanguage("en")
    .Script("latn")
    .Region("gb")
    .Variant("boont")
    .ExtensionsPrefix('r')
    .ExtensionsAdd("extended")
    .ExtensionsAdd("sequence")
    .PrivateUseAdd("private")
    .Build();
languageTag.ToString(); // en-latn-gb-boont-r-extended-sequence-x-private

LanguageTag languageTag = new LanguageTagBuilder()
    .PrivateUseAddRange(["private", "use"])
    .Build();
languageTag.ToString(); // x-private-use

LanguageTag languageTag = new LanguageTagBuilder()
    .Language("ar")
    .ExtendedLanguage("latn")
    .Region("de")
    .Variant("nedis")
    .Normalize();
languageTag.ToString(); // arb-Latn-DE-nedis

Tag Parser

The LanguageTagParser class Parse() method will parse the text form language tag and return a constructed LanguageTag class, or null in case of parsing failure.

Parsing will validate all subtags for correctness in type, length, and position, but not value, and case will not be modified.

Grandfathered tags will be converted to their current preferred form and parsed as such.
E.g. en-gb-oed → en-GB-oxendict, i-klingon → tlh.

The Normalize() method will parse the text tag, and perform validation and normalization. See Tag Normalization for details.

LanguageTag languageTag = new LanguageTagParser()
    .Parse("en-latn-gb-boont-r-extended-sequence-x-private");
// languageTag.Language = en
// languageTag.Script = latn
// languageTag.Region = gb
// languageTag.VariantList = [ boont ]
// languageTag.ExtensionList = [ Prefix: r, TagList: [ extended, sequence ] ]
// languageTag.PrivateUse = [ Prefix: x, TagList: [ private ] ]
languageTag.ToString(); // en-latn-gb-boont-r-extended-sequence-x-private

LanguageTag languageTag = new LanguageTagParser()
    .Parse("en-gb-oed"); // Grandfathered
// languageTag.Language = en
// languageTag.Region = gb
// languageTag.VariantList = [ oxendict ]
languageTag.ToString(); // en-gb-oxendict

Tag Normalization

The LanguageTagParser class Normalize() method will convert tags to their canonical form.
See RFC 5646 Section 4.5 for details

Normalization includes the following:

• Replace the language subtag with their preferred values.
  • E.g. iw → he, in → id
• Replace extended language subtags with their preferred language subtag values.
  • E.g. ar-afb → afb, zh-yue → yue
• Remove or replace redundant subtags their preferred values.
  • E.g. zh-cmn-Hant → cmn-Hant, zh-gan → gan, sgn-CO → csn
• Remove redundant script subtags.
  • E.g. af-Latn → af, en-Latn → en
• Normalize case.
  • All subtags lowercase.
  • Script title case, e.g. Latn.
  • Region uppercase, e.g. GB.
• Sort sub tags.
  • Sort variant subtags by value.
  • Sort extension subtags by prefix and subtag values.
  • Sort private use subtags by value.

languageTag = new LanguageTagBuilder()
    .Language("en")
    .ExtensionAdd('b', ["ccc"]) // Add b before a to force a sort
    .ExtensionAdd('a', ["bbb", "aaa"]) // Add bbb before aaa to force a sort
    .PrivateUseAddRange(["ccc", "a"]) // Add ccc before a to force a sort
    .Normalize();
languageTag.ToString(); // en-a-aaa-bbb-b-ccc-x-a-ccc

LanguageTag languageTag = new LanguageTagParser()
    .Normalize("en-latn-gb-boont-r-sequence-extended-x-private");
languageTag.ToString(); // en-GB-boont-r-extended-sequence-x-private

LanguageTag languageTag = new LanguageTagParser()
    .Parse("ar-arb-latn-de-nedis-foobar");
languageTag.ToString(); // ar-arb-latn-de-nedis-foobar

LanguageTag normalizeTag = new LanguageTagParser()
    .Normalize(languageTag);
normalizeTag.ToString(); // arb-Latn-DE-foobar-nedis

Tag Validation

The LanguageTagParser and LanguageTag class Validate() method will verify subtags for correctness.
See RFC 5646 Section 2.1 and RFC 5646 Section 2.2.9 for details. Refer to Tag Format for a summary.

Note that LanguageTag objects created by Parse() or Normalize() are already verified for form correctness during parsing, and Validate() is primarily of use when using the LanguageTagBuilder Build() method directly.

Validation includes the following:

• Subtag shape correctness, see Tag Format for a summary.
• No duplicate variants, extension prefixes, extension tags, or private tags.
• No missing subtags.

Testing

The BCP47 language subtag lookup site offers convenient tag parsing and validation capabilities.

Refer to unit tests for code validation.
Note that testing attests to the desired behavior in code, but the implemented functionality may not be complete or accurate per the RFC 5646 specification.

References

• Wikipedia : Codes for constructed languages
• Wikipedia : IETF language tag
• W3C : Choosing a Language Tag
• W3C : Language tags in HTML and XML
• W3C : BCP47 language subtag lookup
• IANA : Language Subtags, Tag Extensions, and Tags
• RFC : BCP47
• RFC : 4647 : Matching of Language Tags
• RFC : 5646 : Tags for Identifying Languages
• Unicode Consortium : Unicode Common Locale Data Repository (CLDR) Project
• Library of Congress : ISO 639-2 Language Coding Agency
• SIL International : ISO 639-3 Language Coding Agency

Language Tag Libraries

• github.com/rspeer/langcodes
• github.com/oxigraph/oxilangtag
• github.com/pyfisch/rust-language-tags/
• github.com/DanSmith/languagetags-sharp
• github.com/jkporter/bcp47
• github.com/mattcg/language-subtag-registry

3rd Party Tools

• AwesomeAssertions
• Bring Your Own Badge
• CSharpier
• Create Pull Request
• GH Release
• Git Auto Commit
• GitHub Actions
• GitHub Dependabot
• Husky.Net
• Nerdbank.GitVersioning
• Serilog
• xUnit.Net

License

Licensed under the MIT License