Tokens in Python

Tokens

Tokens are smallest unit of a python program. As we see in chemistry a molecule are made up of combination of elements. Same as we compare in python programming the programs which we write in python are the molecules and the tokens are the elements of python. There are five types of tokens in python:

• Keywords
• Identifiers
• Literals
• Punctuators
• Operators

Keywords

Keywords are nothing but the reserved words which are used by python and have their specific meanings. We are not allowed to use keywords as variables in Python. Keywords in python are case sensitive. Some keywords in python are False, None, True, and, if, class, global, else, finally, for, not, etc.

Identifiers

Identifiers in python are nothing but user defined names to represent variables, functions, classes, etc. But there are few rules that we have to follow while defining an identifier:

• You can define the variable using letters (a to z), numbers (0 to 9) or an underscore (_).
• You cannot use digits to begin an identifier.
• You cannot use reserve words to declare an identifier.
• Other than underscore (_) you cannot use any other special characters, like !, @, #, $, etc. in an identifier.

E.g. of valid identifiers – Roll_no, admin_9, _newvariable, _nv1, _1stud, x123, x, y, z, salary, rollno, etc.
E.g. of invalid identifiers –

• 1student – Starting with digits.
• n%4 – Contains special character(%).
• First name – Contains special character (space).
• None – It is a reserved word.

Literals

Literals can be defined as data that is assigned to a variable or constant. Python supports following literals:

• String Literals
• Numeric Literals
• Boolean Literals
• Special Literals

String Literals

String literals are those texts that you specify under single or double quotation. There are two types of string literals: 
Single Line String Literals – The strings that are terminated within a single line called single line string literals. Eg.

>>>msg = “Hello, how are you?”
>>>print(msg)
Hello, how are you?

Multiline String Literals – A piece of text that is spread along multiline is called as multiline string literals. The multiline string literals can be separated by backslash symbol ( \ ) or by enclosing the text under triple quotation (“”” “”””). You can see the below example:

Example 1:

Example 2:

Numeric Literals

When we assign a value to a variable then that literals are called numeric literals. There are four different numerical types:

• Int (signed integers): Numbers can be both positive and negative with no fractional part. E.g. 145, 90, 1546, etc.
• Long (long integer): Integers of unlimited size followed by lowercase or uppercase L. E.g. 8706537623L, 6537153157263l, etc.
• Float (floating point): Real numbers with both integer and fractional part. E.g. -26.6, 3.14, 10e5 (exponential form), etc.
• Complex (complex): Numbers can be in the form of a+bj where ‘a’ is the real number and ‘bj’ is an imaginary number. E.g. 2 + 17j

Boolean Literals

A Boolean literal have only two values either true or false.

Special Literals

Python contains one special literal i.e. None. None is same as null in other language. We use None to assign null or empty value in some variable. E.g. message=None

Punctuators

These are symbols that are used in programming languages to organize sentence structures. Most common punctuators of Python programming language are:

‘ “ # \ ( ) { } [ ] , : .

Podcast Link: Click Here
YouTube Video Link: Click Here