Note: Dictionary words can contain also slashes when escaped like "/" syntax.
foo Foo/Simpson *bar
In this example, "foo" and "Foo" are personal words, plus Foo will be recognized with affixes of Simpson (Foo's etc.) and bar is a forbidden word.
Dictionary file:
3 hello try/B work/AB
The flags B and A specify attributes of these words.
Affix file:
SET UTF-8 TRY esianrtolcdugmphbyfvkwzESIANRTOLCDUGMPHBYFVKWZ' REP 2 REP f ph REP ph f PFX A Y 1 PFX A 0 re . SFX B Y 2 SFX B 0 ed [^y] SFX B y ied y
In the affix file, prefix A and suffix B have been defined. Flag A defines a `re-' prefix. Class B defines two `-ed' suffixes. First B suffix can be added to a word if the last character of the word isn't `y'. Second suffix can be added to the words terminated with an `y'.
All accepted words with this dictionary and affix combination are: "hello", "try", "tried", "work", "worked", "rework", "reworked".
Hunspell source distribution contains more than 80 examples for option usage.
SET UTF-8
FLAG long
3 hello try/1 work/2
AF definitions in the affix file:
AF 2 AF A AF AB
It is equivalent of the following dic file:
3 hello try/A work/AB
See also tests/alias* examples of the source distribution.
Note I: If affix file contains the FLAG parameter, define it before the AF definitions.
Note II: Use makealias utility in Hunspell distribution to compress aff and dic files.
KEY qwertyuiop|asdfghjkl|zxcvbnm KEY pyfgcrl|aeouidhtns|qjkxbmwvz
Using the first QWERTY layout, Hunspell suggests "nude" and "node" for "*nide". A character may have more neighbors, too:
KEY qwertzuop|yxcvbnm|qaw|say|wse|dsx|sy|edr|fdc|dx|rft|gfv|fc|tgz|hgb|gv|zhu|jhn|hb|uji|kjm|jn|iko|lkm
REP 5 REP f ph REP ph f REP tion$ shun REP ^cooccurr co-occurr REP ^alot$ a_lot
Note I: It's very useful to define replacements for the most typical one-character mistakes, too: with REP you can add higher priority to a subset of the TRY suggestions (suggestion list begins with the REP suggestions).
Note II: Suggesting separated words, specify spaces with underlines:
REP 1 REP onetwothree one_two_three
Note III: Replacement table can be used for a stricter compound word checking with the option CHECKCOMPOUNDREP.
For example a possible mapping could be for the German umlauted ü versus the regular u; the word Frühstück really should be written with umlauted u's and not regular ones
MAP 1 MAP uü
Use parenthesized groups for character sequences (eg. for composed Unicode characters):
MAP 3
MAP ß(ss) (character sequence)
MAP fi(fi) ("fi" compatibility characters for Unicode fi ligature)
MAP (ọ́)o (composed Unicode character: ó with bottom dot)
BREAK 2 BREAK - BREAK -- # n-dash
Breaking are recursive, so foo-bar, bar-foo and foo-foo--bar-bar would be valid compounds. Note: The default word break of Hunspell is equivalent of the following BREAK definition:
BREAK 3 BREAK - BREAK ^- BREAK -$
Hunspell doesn't accept the "-word" and "word-" forms by this BREAK definition:
BREAK 1 BREAK -
Switching off the default values:
BREAK 0
Note II: COMPOUNDRULE is better for handling dashes and other compound joining characters or character strings. Use BREAK, if you want to check words with dashes or other joining characters and there is no time or possibility to describe precise compound rules with COMPOUNDRULE (COMPOUNDRULE handles only the suffixation of the last word part of a compound word).
Note III: For command line spell checking of words with extra characters, set WORDCHARS parameters: WORDCHARS --- (see tests/break.*) example
Note: en_US dictionary of OpenOffice.org uses COMPOUNDRULE for ordinal number recognition (1st, 2nd, 11th, 12th, 22nd, 112th, 1000122nd etc.).
Note II: In the case of long and numerical flag types use only parenthesized flags: (1500)*(2000)?
Note III: COMPOUNDRULE flags work completely separately from the compounding mechanisms using COMPOUNDFLAG, COMPOUNDBEGIN, etc. compound flags. (Use these flags on different entries for words).
The special "endchars" pattern 0 (zero) limits the rule to the unmodified stems (stems and stems with zero affixes):
CHECKCOMPOUNDPATTERN 0/x /y
Note: COMPOUNDMIN doesn't work correctly with the compound word alternation, so it may need to set COMPOUNDMIN to lower value.
(0) Option name (PFX or SFX)
(1) Flag (name of the affix class)
(2) Cross product (permission to combine prefixes and suffixes). Possible values: Y (yes) or N (no)
(3) Line count of the following rules.
Fields of an affix rules:
(0) Option name
(1) Flag
(2) stripping characters from beginning (at prefix rules) or end (at suffix rules) of the word
(3) affix (optionally with flags of continuation classes, separated by a slash)
(4) condition.
Zero stripping or affix are indicated by zero. Zero condition is indicated by dot. Condition is a simplified, regular expression-like pattern, which must be met before the affix can be applied. (Dot signs an arbitrary character. Characters in braces sign an arbitrary character from the character subset. Dash hasn't got special meaning, but circumflex (^) next the first brace sets the complementer character set.)
(5) Optional morphological fields separated by spaces or tabulators.
Note: With CHECKSHARPS declaration, words with sharp s and KEEPCASE flag may be capitalized and uppercased, but uppercased forms of these words may not contain sharp s, only SS. See germancompounding example in the tests directory of the Hunspell distribution.
Hunspell's dictionary items and affix rules may have optional space or tabulator separated morphological description fields, started with 3-character (two letters and a colon) field IDs:
word/flags po:noun is:nom
Example: We define a simple resource with morphological informations, a derivative suffix (ds:) and a part of speech category (po:):
Affix file:
SFX X Y 1 SFX X 0 able . ds:able
Dictionary file:
drink/X po:verb
Test file:
drink drinkable
Test:
$ analyze test.aff test.dic test.txt > drink analyze(drink) = po:verb stem(drink) = po:verb > drinkable analyze(drinkable) = po:verb ds:able stem(drinkable) = drinkable
You can see in the example, that the analyzer concatenates the morphological fields in item and arrangement style.
Marseille ph:maarsayl
feet st:foot is:plural mice st:mouse is:plural teeth st:tooth is:plural
Word forms with multiple stems need multiple dictionary items:
lay po:verb st:lie is:past_2 lay po:verb is:present lay po:noun
sing al:sang al:sung sang st:sing sung st:sing
In affix rules:
SFX Y Y 1 SFX Y 0 ly . ds:ly_adj
In the dictionary:
ably st:able ds:ly_adj able al:ably
feet st:foot is:plural
Useful for zero morphemes and affixes removed by splitting rules.
work/D ts:present
SFX D Y 2 SFX D 0 ed . is:past_1 SFX D 0 ed . is:past_2
Typical example of the terminal suffix is the zero morpheme of the nominative case.
Ispell's original algorithm strips only one suffix. Hunspell can strip another one yet (or a plus prefix in COMPLEXPREFIXES mode).
The twofold suffix stripping is a significant improvement in handling of immense number of suffixes, that characterize agglutinative languages.
A second `s' suffix (affix class Y) will be the continuation class of the suffix `able' in the following example:
SFX Y Y 1 SFX Y 0 s . SFX X Y 1 SFX X 0 able/Y .
Dictionary file:
drink/X
Test file:
drink drinkable drinkables
Test:
$ hunspell -m -d test <test.txt drink st:drink drinkable st:drink fl:X drinkables st:drink fl:X fl:Y
Theoretically with the twofold suffix stripping needs only the square root of the number of suffix rules, compared with a Hunspell implementation. In our practice, we could have elaborated the Hungarian inflectional morphology with twofold suffix stripping.
Hunspell can handle more than 65000 affix classes. There are three new syntax for giving flags in affix and dictionary files.
FLAG long command sets 2-character flags:
FLAG long SFX Y1 Y 1 SFX Y1 0 s 1
Dictionary record with the Y1, Z3, F? flags:
foo/Y1Z3F?
FLAG num command sets numerical flags separated by comma:
FLAG num SFX 65000 Y 1 SFX 65000 0 s 1
Dictionary example:
foo/65000,12,2756
The third one is the Unicode character flags.
Hunspell's dictionary can contain repeating elements that are homonyms:
work/A po:verb work/B po:noun
An affix file:
SFX A Y 1 SFX A 0 s . sf:sg3 SFX B Y 1 SFX B 0 s . is:plur
Test file:
works
Test:
$ hunspell -d test -m <testwords work st:work po:verb is:sg3 work st:work po:noun is:plur
This feature also gives a way to forbid illegal prefix/suffix combinations.
An interesting side-effect of multi-step stripping is, that the appropriate treatment of circumfixes now comes for free. For instance, in Hungarian, superlatives are formed by simultaneous prefixation of leg- and suffixation of -bb to the adjective base. A problem with the one-level architecture is that there is no way to render lexical licensing of particular prefixes and suffixes interdependent, and therefore incorrect forms are recognized as valid, i.e. *legvén = leg + vén `old'. Until the introduction of clusters, a special treatment of the superlative had to be hardwired in the earlier HunSpell code. This may have been legitimate for a single case, but in fact prefix--suffix dependences are ubiquitous in category-changing derivational patterns (cf. English payable, non-payable but *non-pay or drinkable, undrinkable but *undrink). In simple words, here, the prefix un- is legitimate only if the base drink is suffixed with -able. If both these patters are handled by on-line affix rules and affix rules are checked against the base only, there is no way to express this dependency and the system will necessarily over- or undergenerate.
In next example, suffix class R have got a prefix `continuation' class (class P).
PFX P Y 1 PFX P 0 un . [prefix_un]+ SFX S Y 1 SFX S 0 s . +PL SFX Q Y 1 SFX Q 0 s . +3SGV SFX R Y 1 SFX R 0 able/PS . +DER_V_ADJ_ABLE
Dictionary:
2 drink/RQ [verb] drink/S [noun]
Morphological analysis:
> drink drink[verb] drink[noun] > drinks drink[verb]+3SGV drink[noun]+PL > drinkable drink[verb]+DER_V_ADJ_ABLE > drinkables drink[verb]+DER_V_ADJ_ABLE+PL > undrinkable [prefix_un]+drink[verb]+DER_V_ADJ_ABLE > undrinkables [prefix_un]+drink[verb]+DER_V_ADJ_ABLE+PL > undrink Unknown word. > undrinks Unknown word.
Conditional affixes implemented by a continuation class are not enough for circumfixes, because a circumfix is one affix in morphology. We also need CIRCUMFIX option for correct morphological analysis.
# circumfixes: ~ obligate prefix/suffix combinations # superlative in Hungarian: leg- (prefix) AND -bb (suffix) # nagy, nagyobb, legnagyobb, legeslegnagyobb # (great, greater, greatest, most greatest) CIRCUMFIX X PFX A Y 1 PFX A 0 leg/X . PFX B Y 1 PFX B 0 legesleg/X . SFX C Y 3 SFX C 0 obb . +COMPARATIVE SFX C 0 obb/AX . +SUPERLATIVE SFX C 0 obb/BX . +SUPERSUPERLATIVE
Dictionary:
1 nagy/C [MN]
Analysis:
> nagy nagy[MN] > nagyobb nagy[MN]+COMPARATIVE > legnagyobb nagy[MN]+SUPERLATIVE > legeslegnagyobb nagy[MN]+SUPERSUPERLATIVE
Allowing free compounding yields decrease in precision of recognition, not to mention stemming and morphological analysis. Although lexical switches are introduced to license compounding of bases by Ispell, this proves not to be restrictive enough. For example:
# affix file COMPOUNDFLAG X
2 foo/X bar/X
With this resource, foobar and barfoo also are accepted words.
This has been improved upon with the introduction of direction-sensitive compounding, i.e., lexical features can specify separately whether a base can occur as leftmost or rightmost constituent in compounds. This, however, is still insufficient to handle the intricate patterns of compounding, not to mention idiosyncratic (and language specific) norms of hyphenation.
The Hunspell algorithm currently allows any affixed form of words, which are lexically marked as potential members of compounds. Hunspell improved this, and its recursive compound checking rules makes it possible to implement the intricate spelling conventions of Hungarian compounds. For example, using COMPOUNDWORDMAX, COMPOUNDSYLLABLE, COMPOUNDROOT, SYLLABLENUM options can be set the noteworthy Hungarian `6-3' rule. Further example in Hungarian, derivate suffixes often modify compounding properties. Hunspell allows the compounding flags on the affixes, and there are two special flags (COMPOUNDPERMITFLAG and (COMPOUNDFORBIDFLAG) to permit or prohibit compounding of the derivations.
Suffixes with this flag forbid compounding of the affixed word.
We also need several Hunspell features for handling German compounding:
# German compounding # set language to handle special casing of German sharp s LANG de_DE # compound flags COMPOUNDBEGIN U COMPOUNDMIDDLE V COMPOUNDEND W # Prefixes are allowed at the beginning of compounds, # suffixes are allowed at the end of compounds by default: # (prefix)?(root)+(affix)? # Affixes with COMPOUNDPERMITFLAG may be inside of compounds. COMPOUNDPERMITFLAG P # for German fogemorphemes (Fuge-element) # Hint: ONLYINCOMPOUND is not required everywhere, but the # checking will be a little faster with it. ONLYINCOMPOUND X # forbid uppercase characters at compound word bounds CHECKCOMPOUNDCASE # for handling Fuge-elements with dashes (Arbeits-) # dash will be a special word COMPOUNDMIN 1 WORDCHARS - # compound settings and fogemorpheme for `Arbeit' SFX A Y 3 SFX A 0 s/UPX . SFX A 0 s/VPDX . SFX A 0 0/WXD . SFX B Y 2 SFX B 0 0/UPX . SFX B 0 0/VWXDP . # a suffix for `Computer' SFX C Y 1 SFX C 0 n/WD . # for forbid exceptions (*Arbeitsnehmer) FORBIDDENWORD Z # dash prefix for compounds with dash (Arbeits-Computer) PFX - Y 1 PFX - 0 -/P . # decapitalizing prefix # circumfix for positioning in compounds PFX D Y 29 PFX D A a/PX A PFX D Ä ä/PX Ä . . PFX D Y y/PX Y PFX D Z z/PX Z
Example dictionary:
4 Arbeit/A- Computer/BC- -/W Arbeitsnehmer/Z
Accepted compound compound words with the previous resource:
Computer Computern Arbeit Arbeits- Computerarbeit Computerarbeits- Arbeitscomputer Arbeitscomputern Computerarbeitscomputer Computerarbeitscomputern Arbeitscomputerarbeit Computerarbeits-Computer Computerarbeits-Computern
Not accepted compoundings:
computer arbeit Arbeits arbeits ComputerArbeit ComputerArbeits Arbeitcomputer ArbeitsComputer Computerarbeitcomputer ComputerArbeitcomputer ComputerArbeitscomputer Arbeitscomputerarbeits Computerarbeits-computer Arbeitsnehmer
This solution is still not ideal, however, and will be replaced by a pattern-based compound-checking algorithm which is closely integrated with input buffer tokenization. Patterns describing compounds come as a separate input resource that can refer to high-level properties of constituent parts (e.g. the number of syllables, affix flags, and containment of hyphens). The patterns are matched against potential segmentations of compounds to assess wellformedness.
Both Ispell and Myspell use 8-bit ASCII character encoding, which is a major deficiency when it comes to scalability. Although a language like Hungarian has a standard ASCII character set (ISO 8859-2), it fails to allow a full implementation of Hungarian orthographic conventions. For instance, the '--' symbol (n-dash) is missing from this character set contrary to the fact that it is not only the official symbol to delimit parenthetic clauses in the language, but it can be in compound words as a special 'big' hyphen.
MySpell has got some 8-bit encoding tables, but there are languages without standard 8-bit encoding, too. For example, a lot of African languages have non-latin or extended latin characters.
Similarly, using the original spelling of certain foreign names like Ångström or Molière is encouraged by the Hungarian spelling norm, and, since characters 'Å' and 'è' are not part of ISO 8859-2, when they combine with inflections containing characters only in ISO 8859-2 (like elative -ből, allative -től or delative -ről with double acute), these result in words (like Ångströmről or Molière-től.) that can not be encoded using any single ASCII encoding scheme.
The problems raised in relation to 8-bit ASCII encoding have long been recognized by proponents of Unicode. It is clear that trading efficiency for encoding-independence has its advantages when it comes a truly multi-lingual application. There is implemented a memory and time efficient Unicode handling in Hunspell. In non-UTF-8 character encodings Hunspell works with the original 8-bit strings. In UTF-8 encoding, affixes and words are stored in UTF-8, during the analysis are handled in mostly UTF-8, under condition checking and suggestion are converted to UTF-16. Unicode text analysis and spell checking have a minimal (0-20%) time overhead and minimal or reasonable memory overhead depends from the language (its UTF-8 encoding and affixation).
dictionary (xx.cwl -> xx.wl):
preunzip xx.cwl wc -l < xx.wl > xx.dic cat xx.wl >> xx.dic
affix file
If the affix file exists, copy it:
cp xx_affix.dat xx.affIf not, create it with the suitable character encoding (see xx.dat)
echo "SET ISO8859-x" > xx.affor
echo "SET UTF-8" > xx.aff
It's useful to add a TRY option with the characters of the dictionary with frequency order to set edit distance suggestions:
echo "TRY qwertzuiopasdfghjklyxcvbnmQWERTZUIOPASDFGHJKLYXCVBNM" >>xx.aff