Wednesday, January 1, 2020

BigData Python Concepts


BigData Python Concepts

- Simple to understand, Easy to learn
- Supports Interactive mode & Script mode
- Interpreted Language
- Case Sensitive, Dynamically Typed
- Object oriented structure
- Platform independent
- Open source
- It has huge number of libraries/packages
- Very useful in Data Science

Important Packages for BigData programming :
  • NumPy
  • pandas
  • SciPy
  • Statsmodels
  • Scikits
  • matplotlib
  • BeautifulSoup

IPython notebook - IPython is a software package in Python that serves as a research notebook. By using IPython, we can write notes as well as perform data analytics in the same file. i.e., write the code and run it from within the notebook itself.

Indentation 
  • Every line is a new line in the Python. There is no end of the line specifier.
  • Indentation = Whitespace at the beginning of the line.
  • Statements which go together must have same indentation. Each such set of statements is called a block.
Variables and Data Structures : 
Build-in data types : Integer, String, Float, Boolean, Date and Time.
Additional data Structures : Lists, Tuples, Dictionary

String Exercises :
course = 'Python for Beginners'print (course)
print (course[0])
print (course[-1])
print (course[2:5])
print (course[2:])
print (course[-10:])
print (course * 2)
print (course + "TEST")

# Different String functionsprint(course.replace("Python", "Jython"))
print(type(course))
print(len(course))
print(course.upper())
print(course.lower())
print(course.count('n'))
print(course.split(' '))
print(course.swapcase())
print(course.strip())
print(course.lstrip())
print(course.rstrip())
print(":".join(course))
print('B' in course)
print(course.find('g'))

empno = input("Enter EMP NO : ")
print(int(empno) + 10)

Lists - Collection of elements of different data types. 
         - List contains items separated by commas and enclosed within square brackets.

Lists Exercises :
emp_list = ['E100','Ravi',1000.00,'Hyderabad','D100']
dept_list = ['D100','Accounts','Delhi']
print(emp_list)
print(dept_list)
print(emp_list[0])
print(emp_list[2:5])
print(emp_list[2:])
print(dept_list * 2)
print(emp_list + dept_list)

print(len(emp_list))
emp_list.append(25)
emp_list.insert(0,'SNO1')
emp_list.extend(['boy1','boy2'])
print(emp_list.index('E100'))
print(emp_list)
emp_list.remove('boy1')
emp_list.pop(1)
print(emp_list)
employees_age_list = [25,20,35,32,21,28,38,45,23,33]
employees_age_list.sort()
print(employees_age_list)
print(len(employees_age_list))
print(max(employees_age_list))
print(min(employees_age_list))
employees_name_list = ['Ravi','Aditya','Giri','Mohanbabu','Madhu']
print(sorted(employees_name_list, key=len))

Dictionary - Consists of Key-Value pairs. enclosed within curly braces.
                   - Key is any Python type, but are usually numbers or strings.
                   - Values can be an arbitrary Python object.

Dictionary Exercises :
# Dictionary Excersizeemp_dictionary={}
print(emp_dictionary)
emp_dictionary['Eno'] = 'E101'emp_dictionary[2] = 'Ename'dept_dictionary = {'dno':101,'dname':'Account','dloc':'Delhi'}
print(emp_dictionary['Eno'])
print(emp_dictionary[2])
print(dept_dictionary)
print(dept_dictionary.keys())
print(dept_dictionary.values())
# empno = input("Enter EMP NO : ")# print(int(empno) + 10)

 Conditions - Python supports IF-ELSE, FOR, WHILE....conditions.

IF-Else Exercises:
# IF Condition Excerciseemp_age = int(input('Enter Your Age : '))
if(emp_age >= 25 and emp_age <= 30):
    print('You are eligible for Fresher post')
elif(emp_age < 25):
    print('You are not eligible for the Test')
elif(emp_age > 30 and emp_age < 60):
    print('Your are eligible for Experience post')
else:
    print('Invalid Entry')

emp_name_list = ['Ravi','Giri','John']
emp_name = input('Enter Employee Name : ')
if(emp_name in employees_name_list):
    print(f'{emp_name} present in the employees_name_list')
else:
    print(f'{emp_name} is not present in the emp_name_list')

 WHILE Exercises:
# While Condition Excerciseemp_count = 0while(emp_count <= 10):
    print(emp_count)
    emp_count += 1print('End of While loop')

FOR Exercises:
# For Condition Excercisefor dept_no in ('d100','d101','d102'):
    print(dept_no)

for emp_age in range (20,30,2):
    print(emp_age)

emp_age = [22, 24, 25, 32, 35, 41]
total_emp_age = 0for age in emp_age:
    total_emp_age += age
print(total_emp_age)

emp_sno = [1, 3, 5, 7, 8]
sum_emp_sno = [i+2 for i in emp_sno]
print(sum_emp_sno)

get_emp_sno = [i+2 for i in emp_sno if i<5]
print(get_emp_sno)

for i in range(1,10):
    if(i==5):
        break    print(i)
print('Done')

 File System in Python - Python supports a number of formats for file reading and writing.
                                   
- In order to open a file, use open() method specifying file name and mode of opening(read,write,append..etc)
- Open() returns a file handle.
- handle = Open(filename, mode)
- Finally we need to close the file using close() method.(Otherwise other programs might not be able to access the file.

 File Exercises:
# Files - Open & Close Excerciseemp_file = open('C:/Python_Testing/file1.txt','r')
#for line in emp_file:#    print(line)print('File name is ',emp_file.name)
test = emp_file.read()
print(test)
emp_file_out = open('C:/Python_Testing/file2.txt', 'w')
#emp_file_out.seek(0,0)print(emp_file_out.write(test))
emp_file_out.close()
emp_file.close()
We can access any type of file in Python. Example convert SAS file to Text file and access the data. Python has all specific packages for all type of files.
import sas7bdat
from sas7bdat import *
# To convert a SAS file to a text file.
data = SAS7BDAT('C:/Python_Testing/e.sas7bdat')
data.convertFile('C:/Python_Testing/e_data.txt', '\t')



Functions - Reusable piece of software.

Block of statements

- That accepts some arguments. Function accepts any number of arguments.

- Perform some functionality & provide the output.

- define using def keyword.

- Scope of the variables defined inside the function is Local. These variables can't be used outside of a function.



Functions Exercises:


# Function Excercisedef sayHello():
    print('Hello World !!!')
sayHello()

def printMax(a,b):
    if(a > b):
        print(a, 'is Max value')
    elif(a == b):
        print(a, 'is equal to', b)
    else:
        print(b, 'is Max value')
printMax(5,3)

def hello(message, times=1):
    print(message * times)
hello('Welcome', 5)
hello('Hi')

def varfunc(a, b=5, c=10):
    print('a is', a, 'and b is', b, 'and c is', c)
varfunc(3,4)
varfunc(1,2,3)
varfunc(10,c=12)
varfunc(c=15,a=10)

x=20def varscopefunc(x):
    print('x value is ', x)
    x = 2    print('x local value changed to ', x)
varscopefunc(x)
print('x value is not changed', x)



Modules - Functions can be re/used with in the same program. If you want to use functions outside of the programs that can be achieved using Modules.



- Module is nothing but a package of functions.

- Modules can be imported in other programs & functions contained in those Modules can be used.

- Module create - create a .py file with functions defined in that file.

- Python has huge list of Modules, which are pre-defined. We just need to re/use those modules by using import.



Modules Exercises:


def sayHi():
    print('This is mymodule function')
    return# Factorial Programdef factorial(number):
    product = 1    for i in range(number):
        product = product * (i + 1)
    print(product)
    return product
# C:\Users\khasi\AppData\Local\Programs\Python\Python37-32\Lib\mymodule.py






from mymodule import *
print(sayHi())

numb = int(input('Enter a non-negative number : '))
num_factorial = factorial(numb)
print(num_factorial)


Main function:


import sys
# print('First Line')def main():
    print('Hello World!!!', sys.argv[0])
if __name__ == '__main__':
    main()



Exception Handling - Program will terminate abruptly if you don't handle exceptions at run time. Exceptions are handled in Python using Try-Except & Try-Except-Finally blocks.



Finally is an optional block. Finally block will be executed with or with out exception in the program.



Exception Handling Exercises:


def avg(numlist):
    ''' raise TypeError or ZeroDivisionError Exceptions.'''    sum=0    for num in numlist:
        sum = sum + num
    return float(sum)/len(numlist)

def avgReport(numlist):
        try:
            m = avg(numlist)
            print('Avg is =',m)
        except TypeError:
            print('Type Error')
        except ZeroDivisionError:
            print('Zero Division Error')

list1=[10,20,30,40]
list2=[]
list3=[10,20,30,'abc']

avgReport(list1)
print(avgReport(list2))
print(avgReport(list3))






def avg(numlist):
    ''' raise TypeError or ZeroDivisionError Exceptions.'''    sum=0    for num in numlist:
        sum = sum + num
    return float(sum)/len(numlist)

def avgReport(numlist):
        try:
            m = avg(numlist)
            print('Avg is =',m)
        except TypeError:
            print('Type Error')
        except ZeroDivisionError:
            print('Zero Division Error')
        finally:
            print('Finished avg program')

list1=[10,20,30,40]
list2=[]
list3=[10,20,30,'abc']

avgReport(list2)














Posted by



at







































No comments:















Post a Comment


















        

      









Subscribe to:
Post Comments (Atom)