lex [-t] [-n|-v] [file...]
The following options shall be supported:
If the -t option is not specified:
If the -t option is not specified:
When
lex.yy.c
is compiled and linked with the
lex
library (using the
-l l
operand with
c99),
the resulting program shall read character input from the standard
input and shall partition it into strings that match the given
expressions.
When an expression is matched, these actions shall occur:
During pattern matching, lex shall search the set of patterns for the single longest possible match. Among rules that match the same number of characters, the rule given first shall be chosen.
The general format of lex source shall be:
The first "%%" is required to mark the beginning of the rules (regular expressions and actions); the second "%%" is required only if user subroutines follow.
Any line in the Definitions section beginning with a <blank> shall be assumed to be a C program fragment and shall be copied to the external definition area of the lex.yy.c file. Similarly, anything in the Definitions section included between delimiter lines containing only "%{" and "%}" shall also be copied unchanged to the external definition area of the lex.yy.c file.
Any such input (beginning with a <blank> or within "%{" and "%}" delimiter lines) appearing at the beginning of the Rules section before any rules are specified shall be written to lex.yy.c after the declarations of variables for the yylex() function and before the first line of code in yylex(). Thus, user variables local to yylex() can be declared here, as well as application code to execute upon entry to yylex().
The action taken by lex when encountering any input beginning with a <blank> or within "%{" and "%}" delimiter lines appearing in the Rules section but coming after one or more rules is undefined. The presence of such input may result in an erroneous definition of the yylex() function.
C-language code in the input shall not contain C-language trigraphs. The C-language code within "%{" and "%}" delimiter lines shall not contain any lines consisting only of "%}", or only of "%%".
Definitions appear before the first "%%" delimiter. Any line in this section not contained between "%{" and "%}" lines and not beginning with a <blank> shall be assumed to define a lex substitution string. The format of these lines shall be:
name substitute
If a name does not meet the requirements for identifiers in the ISO C standard, the result is undefined. The string substitute shall replace the string {name} when it is used in a rule. The name string shall be recognized in this context only when the braces are provided and when it does not appear within a bracket expression or within double-quotes.
In the Definitions section, any line beginning with a <percent-sign> ('%') character and followed by an alphanumeric word beginning with either 's' or 'S' shall define a set of start conditions. Any line beginning with a '%' followed by a word beginning with either 'x' or 'X' shall define a set of exclusive start conditions. When the generated scanner is in a %s state, patterns with no state specified shall be also active; in a %x state, such patterns shall not be active. The rest of the line, after the first word, shall be considered to be one or more <blank>-separated names of start conditions. Start condition names shall be constructed in the same way as definition names. Start conditions can be used to restrict the matching of regular expressions to one or more states as described in Regular Expressions in lex.
Implementations shall accept either of the following two mutually-exclusive declarations in the Definitions section:
The default type of yytext is implementation-defined. If an application refers to yytext outside of the scanner source file (that is, via an extern), the application shall include the appropriate %array or %pointer declaration in the scanner source file.
Implementations shall accept declarations in the Definitions section for setting certain internal table sizes. The declarations are shown in the following table.
| ||||||||||||||||||||||||
In the table, n represents a positive decimal integer, preceded by one or more <blank> characters. The exact meaning of these table size numbers is implementation-defined. The implementation shall document how these numbers affect the lex utility and how they are related to any output that may be generated by the implementation should limitations be encountered during the execution of lex. It shall be possible to determine from this output which of the table size values needs to be modified to permit lex to successfully generate tables for the input language. The values in the column Minimum Value represent the lowest values conforming implementations shall provide.
The rules in lex source files are a table in which the left column contains regular expressions and the right column contains actions (C program fragments) to be executed when the expressions are recognized.
ERE action ERE action ...
The extended regular expression (ERE) portion of a row shall be separated from action by one or more <blank> characters. A regular expression containing <blank> characters shall be recognized under one of the following conditions:
Anything in the user subroutines section shall be copied to lex.yy.c following yylex().
The lex utility shall support the set of extended regular expressions (see the Base Definitions volume of POSIX.1-2017, Section 9.4, Extended Regular Expressions), with the following additions and exceptions to the syntax:
Within an ERE, a <backslash> character shall be considered to begin an escape sequence as specified in the table in the Base Definitions volume of POSIX.1-2017, Chapter 5, File Format Notation ('\\', '\a', '\b', '\f', '\n', '\r', '\t', '\v'). In addition, the escape sequences in the following table shall be recognized.
A literal
<newline>
cannot occur within an ERE; the escape sequence
'\n'
can be used to represent a
<newline>.
A
<newline>
shall not be matched by a period operator.
| ||||||||||||||||||||||||
The order of precedence given to extended regular expressions for lex differs from that specified in the Base Definitions volume of POSIX.1-2017, Section 9.4, Extended Regular Expressions. The order of precedence for lex shall be as shown in the following table, from high to low.
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
The ERE anchoring operators 'ha' and '$' do not appear in the table. With lex regular expressions, these operators are restricted in their use: the 'ha' operator can only be used at the beginning of an entire regular expression, and the '$' operator only at the end. The operators apply to the entire regular expression. Thus, for example, the pattern "(haabc)|(def$)" is undefined; it can instead be written as two separate rules, one with the regular expression "haabc" and one with "def$", which share a common action via the special '|' action (see below). If the pattern were written "haabc|def$", it would match either "abc" or "def" on a line by itself.
Unlike the general ERE rules, embedded anchoring is not allowed by most historical lex implementations. An example of embedded anchoring would be for patterns such as "(ha| )foo( |$)" to match "foo" when it exists as a complete word. This functionality can be obtained using existing lex features:
hafoo/[ \n] | " foo"/[ \n] /* Found foo as a separate word. */
Note also that '$' is a form of trailing context (it is equivalent to "/\n") and as such cannot be used with regular expressions containing another instance of the operator (see the preceding discussion of trailing context).
The additional regular expressions trailing-context operator '/' can be used as an ordinary character if presented within double-quotes, "/"; preceded by a <backslash>, "\/"; or within a bracket expression, "[/]". The start-condition '<' and '>' operators shall be special only in a start condition at the beginning of a regular expression; elsewhere in the regular expression they shall be treated as ordinary characters.
The action to be taken when an ERE is matched can be a C program fragment or the special actions described below; the program fragment can contain one or more C statements, and can also include special actions. The empty C statement ';' shall be a valid action; any string in the lex.yy.c input that matches the pattern portion of such a rule is effectively ignored or skipped. However, the absence of an action shall not be valid, and the action lex takes in such a condition is undefined.
The specification for an action, including C statements and special actions, can extend across several lines if enclosed in braces:
ERE <one or more blanks> { program statement
program statement }
The program statements shall not contain unbalanced curly brace preprocessing tokens.
The default action when a string in the input to a lex.yy.c program is not matched by any expression shall be to copy the string to the output. Because the default behavior of a program generated by lex is to read the input and copy it to the output, a minimal lex source program that has just "%%" shall generate a C program that simply copies the input to the output unchanged.
Four special actions shall be available:
| ECHO; REJECT; BEGIN
BEGIN newstate;
switches the state (start condition) to newstate. If the string newstate has not been declared previously as a start condition in the Definitions section, the results are unspecified. The initial state is indicated by the digit '0' or the token INITIAL.
The functions or macros described below are accessible to user code included in the lex input. It is unspecified whether they appear in the C code output of lex, or are accessible only through the -l l operand to c99 (the lex library).
The following functions shall appear only in the lex library accessible through the -l l operand; they can therefore be redefined by a conforming application:
Except for input(), unput(), and main(), all external and static names generated by lex shall begin with the prefix yy or YY.
The following sections are informative.
The purpose of input() is to take characters off the input stream and discard them as far as the lexical analysis is concerned. A common use is to discard the body of a comment once the beginning of a comment is recognized.
The lex utility is not fully internationalized in its treatment of regular expressions in the lex source code or generated lexical analyzer. It would seem desirable to have the lexical analyzer interpret the regular expressions given in the lex source according to the environment specified when the lexical analyzer is executed, but this is not possible with the current lex technology. Furthermore, the very nature of the lexical analyzers produced by lex must be closely tied to the lexical requirements of the input language being described, which is frequently locale-specific anyway. (For example, writing an analyzer that is used for French text is not automatically useful for processing other languages.)
%{
/* Need this for the call to atof() below. */
#include <math.h>
/* Need this for printf(), fopen(), and stdin below. */
#include <stdio.h>
%}
DIGIT [0-9]
ID [a-z][a-z0-9]*
%%
{DIGIT}+ {
printf("An integer: %s (%d)\n", yytext,
atoi(yytext));
}
{DIGIT}+"."{DIGIT}* {
printf("A float: %s (%g)\n", yytext,
atof(yytext));
}
if|then|begin|end|procedure|function {
printf("A keyword: %s\n", yytext);
}
{ID} printf("An identifier: %s\n", yytext);
"+"|"-"|"*"|"/" printf("An operator: %s\n", yytext);
"{"[ha}\n]*"}" /* Eat up one-line comments. */
[ \t\n]+ /* Eat up white space. */
. printf("Unrecognized character: %s\n", yytext);
%%
int main(int argc, char *argv[])
{
++argv, --argc; /* Skip over program name. */
if (argc > 0)
yyin = fopen(argv[0], "r");
else
yyin = stdin;
yylex();
}
The current description of lex bypasses the issue of dealing with internationalized EREs in the lex source code or generated lexical analyzer. If it follows the model used by awk (the source code is assumed to be presented in the POSIX locale, but input and output are in the locale specified by the environment variables), then the tables in the lexical analyzer produced by lex would interpret EREs specified in the lex source in terms of the environment variables specified when lex was executed. The desired effect would be to have the lexical analyzer interpret the EREs given in the lex source according to the environment specified when the lexical analyzer is executed, but this is not possible with the current lex technology.
The description of octal and hexadecimal-digit escape sequences agrees with the ISO C standard use of escape sequences.
Earlier versions of this standard allowed for implementations with bytes other than eight bits, but this has been modified in this version.
There is no detailed output format specification. The observed behavior of lex under four different historical implementations was that none of these implementations consistently reported the line numbers for error and warning messages. Furthermore, there was a desire that lex be allowed to output additional diagnostic messages. Leaving message formats unspecified avoids these formatting questions and problems with internationalization.
Although the %x specifier for exclusive start conditions is not historical practice, it is believed to be a minor change to historical implementations and greatly enhances the usability of lex programs since it permits an application to obtain the expected functionality with fewer statements.
The %array and %pointer declarations were added as a compromise between historical systems. The System V-based lex copies the matched text to a yytext array. The flex program, supported in BSD and GNU systems, uses a pointer. In the latter case, significant performance improvements are available for some scanners. Most historical programs should require no change in porting from one system to another because the string being referenced is null-terminated in both cases. (The method used by flex in its case is to null-terminate the token in place by remembering the character that used to come right after the token and replacing it before continuing on to the next scan.) Multi-file programs with external references to yytext outside the scanner source file should continue to operate on their historical systems, but would require one of the new declarations to be considered strictly portable.
The description of EREs avoids unnecessary duplication of ERE details because their meanings within a lex ERE are the same as that for the ERE in this volume of POSIX.1-2017.
The reason for the undefined condition associated with text beginning with a <blank> or within "%{" and "%}" delimiter lines appearing in the Rules section is historical practice. Both the BSD and System V lex copy the indented (or enclosed) input in the Rules section (except at the beginning) to unreachable areas of the yylex() function (the code is written directly after a break statement). In some cases, the System V lex generates an error message or a syntax error, depending on the form of indented input.
The intention in breaking the list of functions into those that may appear in lex.yy.c versus those that only appear in libl.a is that only those functions in libl.a can be reliably redefined by a conforming application.
The descriptions of standard output and standard error are somewhat complicated because historical lex implementations chose to issue diagnostic messages to standard output (unless -t was given). POSIX.1-2008 allows this behavior, but leaves an opening for the more expected behavior of using standard error for diagnostics. Also, the System V behavior of writing the statistics when any table sizes are given is allowed, while BSD-derived systems can avoid it. The programmer can always precisely obtain the desired results by using either the -t or -n options.
The OPERANDS section does not mention the use of - as a synonym for standard input; not all historical implementations support such usage for any of the file operands.
A description of the translation table was deleted from early proposals because of its relatively low usage in historical applications.
The change to the definition of the input() function that allows buffering of input presents the opportunity for major performance gains in some applications.
The following examples clarify the differences between lex regular expressions and regular expressions appearing elsewhere in this volume of POSIX.1-2017. For regular expressions of the form "r/x", the string matching r is always returned; confusion may arise when the beginning of x matches the trailing portion of r. For example, given the regular expression "a*b/cc" and the input "aaabcc", yytext would contain the string "aaab" on this match. But given the regular expression "x*/xy" and the input "xxxy", the token xxx, not xx, is returned by some implementations because xxx matches "x*".
In the rule "ab*/bc", the "b*" at the end of r extends r's match into the beginning of the trailing context, so the result is unspecified. If this rule were "ab/bc", however, the rule matches the text "ab" when it is followed by the text "bc". In this latter case, the matching of r cannot extend into the beginning of x, so the result is specified.
The Base Definitions volume of POSIX.1-2017, Chapter 5, File Format Notation, Chapter 8, Environment Variables, Chapter 9, Regular Expressions, Section 12.2, Utility Syntax Guidelines
Any typographical or formatting errors that appear in this page are most likely to have been introduced during the conversion of the source files to man page format. To report such errors, see https://www.kernel.org/doc/man-pages/reporting_bugs.html .