QuickData.Characters.FSharp 0.3.0

QuickData.Characters.FSharp

Contents

• Overview
• Character Variation
  • Case Conversion with code examples
  • Character Obfuscation with code examples
  • Textualisation Policy
• Character Set
  • Basics with code examples
  • Variations with code examples
  • Conversion and Obfuscation with code examples
  • Sequence Generation with code examples
  • Specials Generation with code examples
• Spelling with code examples
  • Character Spell Policy
• Exception-free Processing
• Discriminated Union Identity Values
• Dependencies
• Usage

You can visit the QuickData.FSharp GitHub repository to report issues, ask questions, or make suggestions. You can also read about the changes across different versions in the release notes there.

Overview

This QuickData.Characters.FSharp package contains the QuickData.Characters.FSharp namespace for F# developers which provides some types, modules, and their related functions for building and manipulating sequences of characters.

All of the sequences generated by the functions in this package are standard F# sequences, and all of the internal processing is done exclusively with standard F# sequences, so they give you all of the benefits of, and follow the same rules as, standard F# sequences.

The types provided are:

• CaseConversion : Defines different ways to convert letter case;
• CharacterObfuscation : Defines different ways to convert one character to one or more other characters;
• CharacterSet : Defines a sequence of characters;
• TextualisationPolicy : Defines different ways in which Normals can be textualised via the textualise function;
• CharacterSpellPolicy : The ways in which a sequence of words can be generated from a sequence of char via the Char.Seq.spell function.

This package also extends the Char module by adding the Char.Seq module (see spelling below).

Note: You can use IntelliSense in your code editor to get more information about each type and function.

Character Variation

Case Conversion

The CaseConversion type defines various ways in which the case of a letter can be changed.

Notes:

1. Case conversion may only affect characters which are defined by System as letters; all other characters - numerals etc. - remain unaffected by case conversion.

2. Determination of the case of the input char is performed via the Char.IsAsciiLetterLower and Char.IsAsciiLetterUpper functions in System.

3. Case conversion is performed via the Char.ToUpperInvariant and Char.toLowerInvariant functions in System.

The cases are:

• NoCaseConversion : No case conversion will be performed - letter out = letter in;
• RandomlyInvertCase
  : The case of the letter may be changed, or not, according to the output from a random number generator;
• RandomlyToUpperCase
  : A lower-case letter can be converted, according to the output from a random number generator, to an upper-case letter;
• RandomlyToLowerCase
  : An upper-case letter can be converted, according to the output from a random number generator, to a lower-case letter;
• AlwaysInvertCase
  : The letter will always have its case changed to the opposite case;
• AlwaysToUpperCase
  : The letter will always be converted to upper-case;
• AlwaysToLowerCase
  : The letter will always be converted to lower-case;
• FirstOnlyToUpperCase : (*) The first letter will be converted to upper-case while the remaining letters will remain as they are;
• TitleCase : (*) The first letter will be converted to upper-case and the remaining letters will be converted to lower-case.

Note: (*) Some of these case conversions only make sense when used with sequences of more than one character. When used with a single character, these conversions will return the original character.

The CaseConversion module defines various functions which work with characters for the purposes of case conversion.

These include:

• displayName : Returns a string describing the case conversion which can be used for display purposes;
• convert : Returns a character which has been passed through the case conversion process;
• Seq.convert : Returns a sequence of characters, each of which has been passed through the case conversion process.

Character Case Conversion Code Examples

let rng = System.Random 1234 // Randomly-chosen seed.

let same = 'a' |> CaseConversion.convert rng CaseConversion.NoCaseConversion // -> 'a'

let upper = 'a' |> CaseConversion.convert rng CaseConversion.AlwaysToUpperCase // -> 'A'

let lower = 'Z' |> CaseConversion.convert rng CaseConversion.AlwaysToLowerCase // -> 'z'

let depends = 'x' |> CaseConversion.convert rng CaseConversion.RandomlyInvertCase // -> Maybe 'x' or 'X'

Character Case Sequence Conversion Examples

let rng = System.Random 345 // Randomly-chosen seed.

let same = 
    seq { 'a' ; 'B' ; 'c' } 
    |> CaseConversion.Seq.convert rng CaseConversion.NoCaseConversion 
    // -> seq { 'a' ; 'B' ; 'c' }

let allUpper = 
    seq { 'a' ; 'B' ; 'c' } 
    |> CaseConversion.Seq.convert rng CaseConversion.AlwaysToUpperCase 
    // -> seq { 'A' ; 'B' ; 'C' }

let mixed = 
    seq { 'a' ; 'B' ; 'C' ; 'd' } 
    |> CaseConversion.Seq.convert rng CaseConversion.RandomlyInvertCase 
    // -> e.g. seq { 'A' ; 'B' ; 'c' ; 'D' }

let title = 
    seq { 'a' ; 'B' ; 'C' ; 'd' } 
    |> CaseConversion.Seq.convert rng CaseConversion.TitleCase 
    // -> seq { 'A' ; 'b' ; 'c' ; 'd' }

Character Obfuscation

The CharacterObfuscation type defines various ways in which one character can be changed to one or more other character(s).

Notes:

1. Character obfuscation only affects characters which are defined in internal tables; all other characters remain unaffected by character obfuscation.

2. Different people/groups have their own ideas about which characters are correct for munging and leeting, but since the aim of these functions is simply to get a sequence of characters, rather than for communication, that's not considered to be relevant here.

3. Obfuscating characters for the purposes of making a password may not make that password more secure. Seek good advice before using this functionality for the creation of passwords.

The cases are:

• NoObfuscation : No obfuscation will be performed - character out = character in;
• Munge : Converts some characters to a different character which looks a bit like the original, e.g. 'B' → '8'; 's' → '5', etc.;
• Leet : Converts some characters to a sequence of one or more different characters, e.g. 'E' → seq { '3' }, or 'K' → seq{ '|' ; '<' }, etc.

The CharacterObfuscation module defines various functions which work with characters for the purposes of character obfuscation.

These include:

• displayName : Returns a string describing the character obfuscation which can be used for display purposes;
• obfuscate : Returns a sequence of char where the original character has (possibly, see above) been obfuscated;
• Seq.obfuscate : Returns a sequence of char where the characters in the source sequence have (possibly, see above) been obfuscated.

Character Obfuscation Code Examples

let same = 's' |> CharacterObfuscation.obfuscate CharacterObfuscation.NoObfuscation // -> 's'
let munge = 's' |> CharacterObfuscation.obfuscate CharacterObfuscation.Munge // -> seq { '5' }
let leet = 'U' |> CharacterObfuscation.obfuscate CharacterObfuscation.Leet // -> seq { '(' ; '_' ; ')' }

Character Sequence Obfuscation Examples

let same = 
    seq { 'a' ; 'c' ; 'e' } 
    |> CharacterObfuscation.Seq.obfuscate CharacterObfuscation.NoObfuscation 
    // -> seq { 'a' ; 'c' ; 'e' }

let munge = 
    seq { 'c' ; 'a' ; 't' } 
    |> CharacterObfuscation.Seq.obfuscate CharacterObfuscation.Munge 
    // -> seq { '(' ; '@' ; '+' }

let leet = 
    seq { 'D' ; 'O' ; 'G' } 
    |> CharacterObfuscation.Seq.obfuscate CharacterObfuscation.Leet 
    // -> seq { '|' ; ')' ; '0' ; '9' }

Textualisation Policy

The TextualisationPolicy type defines cases to be used when textualising a sequence of Normals with the textualise function.

The cases are, for when the Normal value is 'between' two characters:

• UseNearest : Map to the nearest character in the set (uses Math.Round internally and, for many uses, this will be adequate);
• TendDownwards : Map to the earlier of the two characters (uses Math.Floor internally);
• TendUpwards : Map to the later of the two characters (uses Math.Ceiling internally);
• TendOutwards : Map to the character which is earlier when the value is less then 0.5 or later if greater than or equal to 0.5;
• TendInwards : Map to the character which is later when the value is less then 0.5 or earlier if greater than or equal to 0.5.

The difference between these cases can sometimes be subtle so it is recommended that you experiment to see which is best for your particular requirements with the data you are processing.

See the Specials Generation section below for an example of usage.

Notes:

1. See the QuickData.Core.FSharp documentation for more information about the Normal type.

2. See the QuickData.Numbers.FSharp documentation for more information about the Normal.Seq functions which, as well as other things, generate sequences of Normals.

Character Set

The CharacterSet type defines a sequence of characters with some options for case conversion and character obfuscation.

Notes:

1. Control characters, as defined in Char.IsControl, will be omitted from any character set created by the first two functions. If none of the characters are considered usable then the character set will contain an empty sequence of characters.

2. Character sets will be created with no case conversion and no character obfuscation.

The CharacterSet module defines various values and functions which work with character sets, including:

Basics

• fromCharArray : Returns a character set which contains the characters from the input array of char;
• fromString : Returns a character set which contains the characters from the input string;
• toChars : Returns a new sequence of char which contains the characters that are defined in the character set.

Basics Code Examples

let abcdFromCharArray = CharacterSet.fromCharArray [|'a' ; 'b' ; 'c' ; 'd' |]

let xyzFromString = CharacterSet.fromString "xyz"

let characters = 
    xyzFromString 
    |> CharacterSet.toChars 
    // -> seq { 'x' ; 'y' ; 'z' }

Ready-made Character Sets

Some character sets are available to use without you needing to create them, and these are:

Note: These character sets have no case conversion and no character obfuscation.

• lowerCase : A character set containing the English lower-case letters from 'a' to 'z' (inclusive) in ascending aphabetical order;
• upperCase : A character set containing the English upper-case letters from 'A' to 'Z' (inclusive) in ascending aphabetical order;
• numeralsAll : A character set containing the Arabic numerals from '0' to '9' (inclusive) in ascending numerical order;
• numeralsWithoutZero : A character set containing the Arabic numerals from '1' to '9' (inclusive) in ascending numerical order;
• easilyDistinguishable : A character set containing characters which are easily distinguishable, especially when written down;
• lessDistinguishable : A character set containing characters which are not as easily distinguishable as some others, especially when written down;
• forBasicPassword : A character set containing the English lower- and upper-case letters, all Arabic numerals, and the easily-distinguishable characters;
• forTextualisation : A character set containing the English letters 'z' to 'a' then 'A' to 'Z'.

Variations

• append : Returns a new character set containing the characters from both of the input sets;
• combine : Returns a new character set containing the original characters from the first set and the converted and obfuscated characters from the second set;
• distinct : Returns a new character set containing the characters of the original set but with no duplicated characters;
• rev : Returns a new character set containing the characters of the original set in reverse order;
• shuffled : Returns a new character set containing the characters of the original set shuffled in a random order.

Variations Code Examples

// The CharHelpers.seqAsString function is in the QuickData.FSharp namespace.

let firstCharacterSet = CharacterSet.fromString "123" 

let secondCharacterSet = CharacterSet.fromString "abcd" 

let appendedCharacterSet = CharacterSet.append firstCharacterSet secondCharacterSet 

let appendedCharacterList = 
    appendedCharacterSet 
    |> CharacterSet.toChars 
    |> Seq.toList 
    // -> [ '1' ; '2' ; '3' ; 'a' ; 'b' ; 'c' ; 'd' ]

let repeatedCharacterSet = CharacterSet.fromString "qoweertyuuuuiootp" 

let distinctCharacterSet = repeatedCharacterSet |> CharacterSet.distinct

let distinctString = 
    distinctCharacterSet 
    |> CharacterSet.toChars 
    |> CharHelpers.seqAsString 
    // -> "qowertyuip"

let manyAppendedSets = 
    [ "abcd" ; "123" ; "XY"]
    |> List.map CharacterSet.fromString 
    |> List.reduce CharacterSet.append 
    |> CharacterSet.toChars 
    |> CharHelpers.seqAsString 
    // -> "abcd123XY"

Case Conversion and Character Obfuscation Change

• changeCaseConversion : Returns a new character set with the case conversion changed (some convenience functions are also available);
• changeCharacterObfuscation : Returns a new character set with the character obfuscation changed (some convenience functions are also available).

Case Conversion and Character Obfuscation Change Code Examples

// The CharHelpers.seqAsString function is in the QuickData.FSharp namespace.
// Some functions in these examples are explained later.

// Case Conversion.

let rng = System.Random 345 // Randomly-chosen seed.

let randomlyInvertedCase = 
    CharacterSet.lowerCase
    |> CharacterSet.changeCaseConversion CaseConversion.RandomlyInvertCase
    // or |> CharacterSet.randomlyInvertCase 
    |> CharacterSet.cycled rng 
    |> Seq.take 10 
    |> CharHelpers.seqAsString 
    // -> e.g. "aBcDeFGhiJ"

// Character Obfuscation (using shortcut).

let munged = 
    CharacterSet.lowerCase
    |> CharacterSet.changeCharacterObfuscation CharacterObfuscation.Munge 
    // or |> CharacterSet.munge 
    |> CharacterSet.cycled rng 
    |> Seq.take 10 
    |> CharHelpers.seqAsString 
    // -> "@8(63#9#1j"

Sequence Generation

• random : Returns a new sequence of char which is generated by choosing characters randomly from the characters in the set (some convenience functions are also available);
• cycled : Returns a new sequence of char where the characters in the input character set are endlessly cycled (some convenience functions are also available);
• tombola : Returns a new sequence of char where the characters in the input character set are endlessly shuffled (some convenience functions are also available).

Sequence Generation Code Examples

// The CharHelpers.seqAsString function is in the QuickData.FSharp namespace.

let rng = System.Random 1491 // Randomly-chosen seed.

let characterSet = CharacterSet.fromString "abcde"

let randomString = 
    characterSet 
    |> CharacterSet.random rng 12
    |> CharHelpers.seqAsString 
    // -> e.g. "dbeebceccadc"

let cycledString = 
    characterSet 
    |> CharacterSet.cycled rng 
    |> Seq.take 12
    |> CharHelpers.seqAsString 
    // -> "abcdeabcdeab"

let tombolaString = 
    characterSet 
    |> CharacterSet.tombola rng 
    |> Seq.take 12
    |> CharHelpers.seqAsString 
    // -> e.g. "dbacedcbeaed"

let fourPasswords = 
    [ for _ in 1..4 -> 
        CharacterSet.forBasicPassword 
        |> CharacterSet.tombola rng 
        |> Seq.take 16 
        |> CharHelpers.seqAsString ]
        // -> e.g.
        // [ "NqOsjRB8=lfn7vHA"
        //   "Ryun5iUw9&YJKLAo" 
        //   "0ju&>y=i7KXva52t"
        //   "gOY@DPwZKXRvM<1j" ]

Specials Generation

• pinGenerator : Generates numerical PINs of the specified length where the first numeral is never zero;
• textualise : Returns a sequence of char which is a textual representation of the input sequence of Normals mapped to the characters in a character set.

Note: An alternative method of generating PINs is available in the QuickData.Digits.FSharp package via the Digit.Seq.randomViaPolicy function.

The textualise function can use useful if you want to quickly see the 'shape' of numerical data in a compact form, or if you are working in a text-only environment which doesn't have graphing capabilities. It can be used with any character set.

Specials Generation Code Examples

// The CharHelpers.seqAsString function is in the QuickData.FSharp namespace.

let rng = System.Random 681 // Randomly-chosen seed.

let generatePin = CharacterSet.pinGenerator rng 

let sixPins = 
    [ for _ in 1..6 -> 
        generatePin 4 
        |> CharHelpers.seqAsString ]
    // -> e.g. [ "5240" ; "2547" ; "4749" ; "6135" ; "3179" ; "9849" ]

open QuickData.Numbers.FSharp // To get the sine wave sequence of Normal.

let text = 
    Normal.Seq.byEquation NormalEquation.SineWave 40 
    |> CharacterSet.textualise CharacterSet.forTextualisation TextualisationPolicy.UseNearest 
    |> CharHelpers.seqAsString
    // -> "AEILPSUXYZZYXWTQNJGCcgjnqtwxyzzyxuspliea"

Spelling

The Char.Seq.spell function returns a sequence of string, each element of which may be the spelt version of an input char.

For example, a sequence of 'a', '6', 'c' could produce a sequence of "alfa", "six", "charlie".

Note: The only language available is English. There are no plans to support other languages.

Character Spell Policy

The CharacterSpellPolicy type defines cases to be used when creating a list of strings ('phrases') with the Char.Seq.spell function.

The cases, which specify the types of phrases which can be created, are:

• CivilAviation : As defined by the International Civil Aviation Organization;
• NatoPronunciation : The NATO pronunciation of the characters;
• InternationalMaritime : The International Maritime Organization characters;
• PgpTwoSyllables : From the PGP biometric word list (two syllables);
• PgpThreeSyllables : From the PGP biometric word list (three syllables);
• ClaphamDwyer : From a parody song called "A Surrealist Alphabet" about Cockneys (just a bit of fun and probably not much use, but it's there if you want to use it).

Note: See the Intellisense documentation to see the usable characters for each policy.

Spelling Code Examples

let rng = System.Random 4567 // Randomly-chosen seed.

let words = 
    CharacterSet.lowerCase 
    |> CharacterSet.append CharacterSet.numeralsAll 
    |> CharacterSet.tombola rng 
    |> Seq.take 12 
    // -> e.g. seq { 'n'; 'v'; '3'; 'e'; '8'; 'p'; 'b'; '5'; 'z'; 'i'; 'y'; '4' }
    |> Char.Seq.spell CharacterSpellPolicy.CivilAviation 
    // -> e.g. seq { "november" ; "victor" ; "three" ; "echo" ; "eight" ; "papa"; 
    //               "bravo" ; "five" ; "zulu" ; "india" ; "yankee" ; "four" } 

let quickData = 
    "QuickData.FSharp" // A string is a sequence of char.
    |> Char.Seq.spell CharacterSpellPolicy.PgpTwoSyllables
    // -> seq { "drunken"; "hockey"; "frighten"; "flatfoot"; "geiger"; "crumpled";
    //          "fallout"; "highchair"; "fallout"; "buzzard"; "cubic"; "dwelling"; 
    //          "freedom"; "fallout"; "guidance"; "goldfish" } 

Exception-free Processing

Some functions will raise an exception if the internal processing fails for some reason or if the input was not as expected.

Many of this type of function will also have alternative exception-free versions which do not raise an exception.

These alternative functions are:

• <module-name>.FailSafe.<function-name> : Returns a default value, instead of raising an exception;
• <module-name>.Option.<function-name> : Returns None instead of raising an exception, otherwise Some value;
• <module-name>.Result.<function-name> : Returns Error <error-type> instead of raising an exception, otherwise Ok value.

For example, CaseConversion.fromInt will raise an exception upon failure whereas CaseConversion.Failsafe.fromInt will return a default value (in this case the default case conversion).

Discriminated Union Identity Values

Where a discriminated union exists to specify a parameter for a function, there often will be two functions related to that discriminated union which return an integer identity value from the union case - toInt - or return a union case from an integer identity value - fromInt.

These can be useful if you need to store a value in a data structure which does not cater for discriminated unions, e.g. in a file, in JSON, etc.

The toInt function will always return a valid identity value.

Notes:

1. Where a fromInt function exists there often will be exception-free versions available.

2. All valid identity values are in the range 100 to 999 (inclusive). Any value which has fewer, or more, than three digits can be immediately identified as being invalid, but not all three-digit values are valid. While identity values are unique within a discriminated union, some identity values may be shared by cases in different discriminated unions but no functional equality or relationship between the two should be inferred. Identity values will not change unless specifically mentioned in the release notes.

The defaultCase value (where available) contains the default case for that discriminated union.

The allCases value (where available) contains a list of all of the cases for the discriminated union.

Discriminated Union Identity Value Examples

let good = TextualisationPolicy.fromInt 200 // -> TendDownwards
let bad = TextualisationPolicy.fromInt 123 // Raises an exception

let goodFailSafe = TextualisationPolicy.FailSafe.fromInt 300 // -> TendUpwards
let badFailSafe = TextualisationPolicy.FailSafe.fromInt 123 // -> UseNearest (default)

let goodOption = TextualisationPolicy.Option.fromInt 400 // -> TendOutwards
let badOption = TextualisationPolicy.Option.fromInt 123 // None

let goodResult = TextualisationPolicy.Result.fromInt 100 // -> Ok UseNearest
let badResult = TextualisationPolicy.Result.fromInt 123 // -> Error (InvalidArgumentError ("identity", "123"))

Dependencies

This package is dependent upon FSharp.Core which you will be using anyway, and the QuickData.Core.FSharp package which will normally be automatically installed if you install this package.

Usage

The types and functions in this package have been designed to be used only with F#.

However, they may also be usable with C# but this has not been tested, so use them with C# at your own risk.