Lecture 04: Lexical Analysis – Regular Expressions

What are the components of a language

Lexical Structures, Syntax, Semantics

What is meant by lexical Structures

Its lexical structures, which concerns the forms of its individual symbols (e.g.:=), keywords, identifiers etc.

What is meant by syntax

Its syntax, which define the structure of the components of the language, e.g., the structures of programs, statements (e.g., assignment), expression, terms etc.

What is meant by semantics

Its semantics, which define the meanings and usage of structures and requirements that cannot be describe by a grammar.

Language analysis constist of what 2 parts?

[1] A low-level part called a lexical analyzer (mathematically, a finite automaton based on a regular grammar) [2] A high-level part called a syntax analyzer, or parser (mathematically, a push-down automaton based on a context-free grammar)

What is lexical Analysis

A lexical analyser (scanner) reads the source program, a character at a time, and outputs tokens to the next phase of the compiler (parser)

What does a lexical anayser do?

[1] It identifies substrings of the source program that belong together - lexemes [2] Lexemes match character patterns, which are associated with a lexical category called a token [3] sum is a lexeme; its token may be IDENT

What is RE short for?

regular expressions

What is DFA short for?

deterministic finite automata

Name a use for DFAs

deterministic finite automata (DFAs), which can be used to implement a pattern matching process;

What is the definition of Alphabet

[1] An alphabet Σ is a finite non-empty set(of symbols) eg:[2] set Σab={a, b} is an alphabet comprising symbols a and b; [3] the set Σaz = {a, ..., z} is the alphabet of lowercase English letters; [4] the set Σasc of all ASCII characters is an alphabet.

What is the definition of Strings

[1] A string or word over an alphabet Σ is a finite concatenation (or juxtaposition) of symbols from Σ. [2] abba, aaa and baaaa are strings over Σab; [3] hello, abacab, and baaaa are strings over Σaz; [4] h$(e′lo, PjM#;, and baaaa are strings over Σasc.

How are the lengths of Strings denoted?

[1] The length of a string w (that is, the number of symbols it has) is denoted |w|. E.g., |abba| = 4. [2] The empty or null string is denoted ε, and so |ε| = 0.

What isThe set of all strings over Σ

Σ∗ab = {ε, a, b, aa, ab, ba, bb, aab,...}.

How is the concatenation of a string denoted?

For any symbol or string x, xn denotes the string of the concatenation of n copies of x. E.g. a4 = aaaa (ab)4 = abababab

Regular expressions s__ p___ of strings of symbols

Regular expressions specify patterns of strings of symbols

Regular expressions ___ ____ of strings of ___

Regular expressions specify patterns of strings of symbols

How is the set of Strings matched by RE r denoted?

The set of strings matched by a RE r is denoted L(r) ⊆ Σ∗ (all the strings over the alphabet Σ) and is called the language determined or generated by r

When does a regular expression match a set of strings?

We say a regular expression r matches (or is matched by) a set of strings if the patterns of the strings are those specified by the regular expression.

What is the regular expression ∅

∅ (the symbol for empty-set or empty language, i.e., the set contains nothing) is a regular expression. This RE matches no strings at all

what is the regualr expression ε

ε (the empty string symbol) is a regular expression. This matches just the empty string ε.

what does this mean "Each symbol c ∈ Σ in the alphabet Σ i".

This RE matches the string consisting of just the symbol c.

explain in words Σ = {a, b}

[1] a which matches the string a; and [2] b which matches the string b, [3] Both symbols a and b are REs.

If r and s are regular expressions is r | s a regualr expression?

yes

what are the otehr wars to write r | s

r + s, and read “r or s”)

explain in words a | b

Regular expression a | b matches the strings a or b.

If r and s are regular expressions waht is the concatenation of r and s?

rs(read “r followed by s”)

can brackets be used in regular expressions?

yes, As with arithmetic expressions, parentheses can be used in REs to make the meaning of a regular expression clear.

is r* a regular expression?

yes (read “zero or more instances of r”)

what is r* in words

r∗ (read “zero or more instances of r”) is a RE. This matches all finite (possibly empty) concatenations of strings matched by r.

what is rr*

rr∗ (one or more copies of strings matched by r)

write rr* another way

r^+ (the ^ is her only for computer on paper its just r to the plus)

What does the Regular expression a∗ matches

strings ε, a, aa, aaa,...

What does RE (ab)∗ match

the strings ε, ab, abab,...

what does RE (a|bb)∗ match

the strings ε, a, bb, abb, bba, abba,...

what does RE (a|b)∗aab match

any string ending with aab

What does RE (a|b)∗baa(a|b)∗ match

any string containing the substring bba.

What is the precedence order for regular expressions

[1]() has the highest precedence; [2] then ∗ (or +); [3] then concatenation; and [4] | has lowest precedence.

how are REs used for Lexical Analysis

Regular expressions provide us with a way to describe the patterns of a programming language.

assuming the alphabet is Σasc (the set of all ASCII characters) what is a typical programs pattern(regular expression) for an IF;

if for a token IF;

assuming the alphabet is Σasc (the set of all ASCII characters) what is a typical programs pattern(regular expression) for a;

; for a token SEMICOLON;

assuming the alphabet is Σasc (the set of all ASCII characters) what is a typical programs pattern(regular expression) for (0|1|2|3|4|5|6|7|8|9)+

(0|1|2|3|4|5|6|7|8|9)+ for a token NUMBER;

assuming the alphabet is Σasc (the set of all ASCII characters) what is a typical programs pattern(regular expression) for (a|...|z|A|···|Z)(_|a|...|z|A|···|Z|0|···|9)∗

(a|...|z|A|···|Z)(_|a|...|z|A|···|Z|0|···|9)∗ for a token IDENT.

We can give REs ____ which make REs more easy to read and write, and can be used to define other regular definitions.

We can give REs names, which make REs more easy to read and write, and can be used to define other regular definitions.

what are some examples of named regualr expressions

[1] letter = A | B |···|Z | a | b |···| z [2] digit = 0 | 1 | ··· | 9 [3] ident=letter (_| letter | digit)∗

what is the formal lexical definition of a language

A language L over an alphabet Σ is a subset of Σ∗ (i.e. L ⊆ Σ∗).

using the lexical definition fo a language what is {ε, aab, bb}

{ε, aab, bb} is a language over Σab;

using the lexical definition fo a language what is the set of all Java programs

the set of all Java programs is a language over Σasc;

using the lexical definition fo a language what is ∅

∅ is the empty language (over any alphabet) with no strings;

using the lexical definition fo a language what is {ε}

{ε} is a language (over any alphabet) containing just the empty string.

using the lexical definition fo a language what is Σ∗

Σ∗ is a language over Σ for any alphabet Σ.

How is the language of REs denoted?

L(RE) e.g. (a∗) = {ε, a, aa, aaa,...}

interpreter L(a|b)

L(a|b) = L(a)∪L(b)

explain a decision procedure for L

an algorithm such that a language L over some alphabet Σ is able to take any input stringw ∈ Σ∗, and: 1. outputs ‘Yes’ if w ∈ L and 2. outputs ‘No’ if w not ∈ L.

Languages that can be denoted by a RE, and can have a DFA/NFA as a decision procedure, are known as _____ ______.

Languages that can be denoted by a RE, and can have a DFA/NFA as a decision procedure, are known as regular languages.

Study has shown that we can write a decision procedure for language L(r) using one of the what 2 algorithms?

[1] a Deterministic Finite Automaton (DFA), or [2] a Nondeterministic Finite Automaton (NFA).

What is Nondeterministic Finite Automaton (NFA).

An NFA is a Nondeterministic Finite Automaton. Nondeterministic means it can transition to, and be in, multiple states at once (i.e. for some given input).

what is a Deterministic Finite Automaton (DFA),

A DFA is a Deterministic Finite Automaton. Deterministic means that it can only be in, and transition to, one state at a time (i.e. for some given input).