A compiler or interptreter for a programminning language is often decomposed into two parts:

Read the source program and discover its structure.

Process this structure, e.g. to generate the target program.

Lex and Yacc can generate program fragments that solve the first task.

The task of discovering the source structure again is decomposed into subtasks:



Split the source file into tokens (Lex).

Find the hierarchical structure of the program (Yacc).-Lex helps write programs whose control flow is directed by instances of regular expressions in the input stream. It is well suited for editor-scr ipt type transformations and for segmenting input in preparation for a parsing routine.



Lex source is a table of regular expressions and corresponding program fragments. The table is translated to a program which reads an input stream, copying it to an output stream and partitioning the input into strings which match the given expressions. As each such string is recognized the corresponding program fragment is executed. The recognition of the expressions is performed by a deterministic finite automaton generated by Lex. The program fragments written by the user are executed in the order in which the corresponding regular expressions occur in the input stream.



Computer program input generally has some structure　in fact, every computer program that does input can be thought of as defining an ``input language　 which it accepts. An input l