Lexical Analyzer in C Programming

 

Let us learn about lexical analyzer in C programming and understand how lexical analysis works in C programming with an example.

What is Lexical Analysis?

Lexical analysis occurs at the very first phase of the compilation process. It is also very popularly known as tokenization, and this leads to the efficiency of programming.

 

Lexical analysis is the process of converting the sequence of characters in a source code into a set of tokens. Some of the common tokens are enlisted below:

Token NameExample
Keywordauto, int, if
Identifierx, temp, arr
Separator, ; () {}
Literal/Constants“3.45677”

A lexer is usually combined with a parser to scan the source code to generate the tokens. It works closely with the syntax analyser.

The lexical analyzers help to find the tokens within a given C program and also calculate the total number of tokens present in it.

Must Read: C Program For First and Follow of Grammar

There are some elements that cannot be categorized into tokens which are as follows:

  • Pre-processor directives
  • Macro
  • Blank
  • Newline
  • Tabs
  • Comments

The following lexical analyzer program in C language includes a function that enlists all the keywords available in the C programming library. Lexical analysis is used in compiler designing process.

   

Also, we have only used the arithmetic operators in arithmetic_operators variable which can be modified to include other operators in C programming such as relational, logical, ternary and other operators.

Functions of Lexical Analyser in Compiler Design

  1. Divide the source code into different tokens
  2. Remove comments
  3. Remove whitespaces
  4. Error generation for invalid tokens with line and row numbers.

Lexical Analysis Example

Source code

Result of Lexical Analysis

Identifier: includestdioh

   

Keyword:  int

Identifier: mainvoid

Identifier: printfHello

Identifier: Worldn

Keyword:  return

Identifier: 0

Note: This simple C program to implement lexical analysis is compiled with GNU GCC compiler using CodeLite IDE on Microsoft Windows 10 operating system.

Implementation of Lexical Analyzer in C Programming

Output

simple code to implement lexical analyzer in C programming with explanation and output

If you have any doubts about the implementation of a lexical analyzer in C programming, let us know about it in the comment section.

Let's Discuss