P#18 EDA

Exploratory Data Analysis

For the quick overview we can use following methods and attributes of a DataFrame: df

df.head() # show first 5 rows

df.tail() # last 5 rows

df.columns # list all column names

df.shape # get number of rows and columns

df.info() # additional info about dataframe

df.describe() # statistical description, only for numeric values

df['col_name'].value_counts(dropna=False) # count unique values in a column

import pandas as pd
import matplotlib.pyplot as plt

df = pd.read_csv('iris_csv.csv')
# print(df)
"""
    sepallength  sepalwidth  petallength  petalwidth           class
0            5.1         3.5          1.4         0.2     Iris-setosa
1            4.9         3.0          1.4         0.2     Iris-setosa
2            4.7         3.2          1.3         0.2     Iris-setosa
3            4.6         3.1          1.5         0.2     Iris-setosa
4            5.0         3.6          1.4         0.2     Iris-setosa
..           ...         ...          ...         ...             ...
145          6.7         3.0          5.2         2.3  Iris-virginica
146          6.3         2.5          5.0         1.9  Iris-virginica
147          6.5         3.0          5.2         2.0  Iris-virginica
148          6.2         3.4          5.4         2.3  Iris-virginica
149          5.9         3.0          5.1         1.8  Iris-virginica

[150 rows x 5 columns]
 """
# print(df.head()) # show first 5 rows
"""
   sepallength  sepalwidth  petallength  petalwidth        class
0          5.1         3.5          1.4         0.2  Iris-setosa
1          4.9         3.0          1.4         0.2  Iris-setosa
2          4.7         3.2          1.3         0.2  Iris-setosa
3          4.6         3.1          1.5         0.2  Iris-setosa
4          5.0         3.6          1.4         0.2  Iris-setosa
"""
# print(df.tail()) # last 5 rows
"""
    sepallength  sepalwidth  petallength  petalwidth           class
145          6.7         3.0          5.2         2.3  Iris-virginica
146          6.3         2.5          5.0         1.9  Iris-virginica
147          6.5         3.0          5.2         2.0  Iris-virginica
148          6.2         3.4          5.4         2.3  Iris-virginica
149          5.9         3.0          5.1         1.8  Iris-virginica
"""
#print(df.columns) # list all column names
#Index(['sepallength', 'sepalwidth', 'petallength', 'petalwidth', 'class'], dtype='object')

# print(df.shape) # get number of rows and columns # (150, 5)

# print(df.info()) # additional info about dataframe
"""
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 150 entries, 0 to 149
Data columns (total 5 columns):
 #   Column       Non-Null Count  Dtype  
---  ------       --------------  -----  
 0   sepallength  150 non-null    float64
 1   sepalwidth   150 non-null    float64
 2   petallength  150 non-null    float64
 3   petalwidth   150 non-null    float64
 4   class        150 non-null    object 
dtypes: float64(4), object(1)
memory usage: 6.0+ KB
None
"""
# print(df.describe()) # statistical description, only for numeric values
"""
       sepallength  sepalwidth  petallength  petalwidth
count   150.000000  150.000000   150.000000  150.000000
mean      5.843333    3.054000     3.758667    1.198667
std       0.828066    0.433594     1.764420    0.763161
min       4.300000    2.000000     1.000000    0.100000
25%       5.100000    2.800000     1.600000    0.300000
50%       5.800000    3.000000     4.350000    1.300000
75%       6.400000    3.300000     5.100000    1.800000
max       7.900000    4.400000     6.900000    2.500000
"""
# df['col_name'].value_counts(dropna=False) #

#Another way to quickly check the data is by visualizing it.
#We use bar plots for discrete data counts
#and histogram for continuous.

y = df['sepalwidth']
plt.hist(y)
plt.savefig('eda1.png')
plt.show()

#
df.boxplot(column='sepallength', by='class')
plt.savefig('eda2.png')
plt.show()

## Scatter plot to depict relationship between two variables and show outliers if any

df.plot(kind='scatter', x='sepallength', y='class')
plt.savefig('edascatter.png')
plt.show()

Histogram and box plot can help to spot visually the outliers. The scatter plot shows relationship between 2 numeric variables

To check correlation between variable.

import seaborn as sns
plt.figure(figsize=(8,4))
sns.heatmap(df.corr(),cmap='Reds',annot=False)
plt.savefig('heatmap.png')
plt.show()

Above, positive correlation is represented by dark shades and negative correlation by lighter shades. Changes the value of annot=True, and the output will show you values by which features are correlated to each other in grid-cells.

k = 12
cols = df.corr().nlargest(k, 'sepallength')['sepallength'].index
cm = df[cols].corr()
plt.figure(figsize=(8,6))
sns.heatmap(cm, annot=True, cmap = 'viridis')
plt.savefig('sepallength.png')
plt.show()

From this figure we infer strong correlation between pedalwidth and pedallength since it has maximum + value. Funny! pennywise # Foolish!!!

We can check from heatmap, strong and weak correlation of all variable with their counterparts as shown above.

Happy learning at AMET -ODL!!!

Pandas #05 Pivot Tables

PIVOT TABLE

A pivot table is a similar operation that is commonly seen in spreadsheets and other programs that operate on tabular data. 

The pivot table takes simple column-wise data as input, and groups the entries into a two-dimensional table that provides a multidimensional summarization of the data. 

The difference between pivot tables and GroupBy can sometimes cause confusion; it helps me to think of pivot tables as essentially a multidimensional version of GroupBy aggregation. 

That is, you split-apply-combine, but both the split and the combine happen across not a one-dimensional index, but across a two-dimensional grid.

We will take one sample dataset birth.csv as below:

import pandas  as pd

births = pd.read_csv('https://raw.githubusercontent.com/jakevdp/data-CDCbirths/master/births.csv ')

print(births)
"""
      year  month  day gender  births
0      1969      1  1.0      F    4046
1      1969      1  1.0      M    4440
2      1969      1  2.0      F    4454
3      1969      1  2.0      M    4548
4      1969      1  3.0      F    4548
...     ...    ...  ...    ...     ...
15542  2008     10  NaN      M  183219
15543  2008     11  NaN      F  158939
15544  2008     11  NaN      M  165468
15545  2008     12  NaN      F  173215
15546  2008     12  NaN      M  181235

[15547 rows x 5 columns]

"""

How to count male and female in a decade. So now we have to create column say decade and groupby Gender, and sum those rows.

births['decade'] = 10 * (births['year'] // 10)  # function
births.pivot_table('births', index='decade', columns='gender', aggfunc='sum')
print(births.pivot_table('births', index='decade', columns='gender', aggfunc='sum'))
"""
gender         F         M
decade                    
1960     1753634   1846572
1970    16263075  17121550
1980    18310351  19243452
1990    19479454  20420553
2000    18229309  19106428

"""

Have a deep breath. Look at the output. very easily found that male births outnumbered female births in all decades. How we will plot to explore visually

Cross Tab ( Contingency Table)

A contingency table is a tabular representation of categorical data . A contingency table usually shows frequencies for particular combinations of values of two discrete random variable s X and Y. Each cell in the table represents a mutually exclusive combination of X-Y values.

Used to summarize large data set.

import pandas as pd
import matplotlib.pyplot as plt

df = pd.read_csv('indian_food.csv') # From Kaggle
# print(df)
"""
               name  ...      region
0        Balu shahi  ...        East
1            Boondi  ...        West
2    Gajar ka halwa  ...       North
3            Ghevar  ...        West
4       Gulab jamun  ...        East
..              ...  ...         ...
250       Til Pitha  ...  North East
251         Bebinca  ...        West
252          Shufta  ...       North
253       Mawa Bati  ...     Central
254          Pinaca  ...        West

[255 rows x 9 columns]

"""
#print(df.describe())
#print(df.shape)
#print(df.columns)


# Cross tab
# Compute a simple cross-tabulation of two (or more) factors.
# By default computes a frequency table of the factors unless an array of values and
# an aggregation function are passed.
# implementing crostab on state & diet columns

print(pd.crosstab(df['state'], df['diet']))
"""
diet             non vegetarian  vegetarian
state                                      
-1                            0          24
Andhra Pradesh                0          10
Assam                        10          11
Bihar                         0           3
Chhattisgarh                  0           1
Goa                           1           2
Gujarat                       0          35
Haryana                       0           1
Jammu & Kashmir               0           2
Karnataka                     0           6
Kerala                        1           7
Madhya Pradesh                0           2
Maharashtra                   2          28
Manipur                       1           1
NCT of Delhi                  1           0
Nagaland                      1           0
Odisha                        0           7
Punjab                        4          28
Rajasthan                     0           6
Tamil Nadu                    1          19
Telangana                     1           4
Tripura                       1           0
Uttar Pradesh                 0           9
Uttarakhand                   0           1
West Bengal                   5          19
"""



print(pd.crosstab(df['region'], df['diet']))
"""
diet        non vegetarian  vegetarian
region                                
-1                       0          13
Central                  0           3
East                     5          26
North                    5          44
North East              13          12
South                    3          56
West                     3          71
"""

print(pd.crosstab(df['region'], df['diet'], normalize='all'))  # Note Normalization

"""
diet        non vegetarian  vegetarian
region                                
-1                0.000000    0.051181
Central           0.000000    0.011811
East              0.019685    0.102362
North             0.019685    0.173228
North East        0.051181    0.047244
South             0.011811    0.220472
West              0.011811    0.279528
"""

print(pd.crosstab(df['region'], df['diet'], normalize='index')) # index normalization

# Plotting
pd.crosstab(df['region'], df['diet']).plot(kind='line')

plt.savefig('regionline.png') 

plt.show()

plt.show()

pd.crosstab(df['region'], df['diet']).plot(kind='bar')
plt.savefig('regionbar.png')
plt.show()

pd.crosstab(df['region'], df['diet']).plot(kind='barh')
plt.savefig('regionbarh.png')
plt.show()

print(pd.crosstab(df['flavor_profile'], df['diet']).count)
"""
<bound method DataFrame.count of diet            non vegetarian  vegetarian
flavor_profile
-1                           3          26
bitter                       0           4
sour                         0           1
spicy                       26         107
sweet                        0          88>
"""

crosstab has many uses. Here we discussed about very popular usage of crosstab. 

Pandos #04 -Data Aggregation

 AGGREGATE

Data Frame Aggregation

Python has several methods are available to perform aggregations on data. It is done using the pandas and numpy libraries. Data Frame method support data aggregation. Let us see how we can apply:

import pandas as pd
import numpy as np

df = pd.DataFrame([[1, 2, 3, 4, 5],
                   [4, 5, 6, 7, 8],
                   [7, 8, 9, 10, 11],
                   [np.nan, np.nan, np.nan,np.nan,np.nan]],
                  columns=['A', 'B', 'C', 'D', 'E'])

# over rows
dfagg = df.agg(['sum', 'min'])

print(dfagg)

"""
       A     B     C     D     E
sum  12.0  15.0  18.0  21.0  24.0
min   1.0   2.0   3.0   4.0   5.0
"""

Aggregating different aggregates over columns

# Different aggregate functions in columns

df.agg({'A' : ['sum', 'min'], 'B' : ['min', 'max']})
print(df.agg({'A' : ['sum', 'min'], 'B' : ['min', 'max']}))
"""
       A    B
sum  12.0  NaN
min   1.0  2.0
max   NaN  8.0
"""

This describe() method display all statistical properties like min, max, mean, std, 25%, 50%, 75% quartiles.

print(df.describe())

"""
        A    B    C     D     E
count  3.0  3.0  3.0   3.0   3.0
mean   4.0  5.0  6.0   7.0   8.0
std    3.0  3.0  3.0   3.0   3.0
min    1.0  2.0  3.0   4.0   5.0
25%    2.5  3.5  4.5   5.5   6.5
50%    4.0  5.0  6.0   7.0   8.0
75%    5.5  6.5  7.5   8.5   9.5
max    7.0  8.0  9.0  10.0  11.0
"""

Transformation and manipulation on elements are very easy. Let us see some code snippets

Let us assume we want add +1 to all the above elements.

print(df.transform(lambda x: x + 1))
"""
     A    B     C     D     E
0  2.0  3.0   4.0   5.0   6.0
1  5.0  6.0   7.0   8.0   9.0
2  8.0  9.0  10.0  11.0  12.0
3  NaN  NaN   NaN   NaN   NaN
"""

We an use groupby() too.

df = pd.DataFrame({
    "Date": [
        "2019-05-08", "2019-05-07", "2019-05-06", "2019-05-05",
        "2019-05-08", "2019-05-07", "2019-05-06", "2019-05-05"],
    "Data": [5, 8, 6, 1, 50, 100, 60, 120],
})
print(df)
"""
        Date  Data
0  2019-05-08     5
1  2019-05-07     8
2  2019-05-06     6
3  2019-05-05     1
4  2019-05-08    50
5  2019-05-07   100
6  2019-05-06    60
7  2019-05-05   120
"""

print(df.groupby('Date')['Data'].transform('sum'))

"""
0     55
1    108
2     66
3    121
4     55
5    108
6     66
7    121
Name: Data, dtype: int64
"""

We can groupby different levels of a hierarchical index using the level parameter: Please note the usage of pd.MultiIndex.from.arrays()  Method. So many ways, you can group by multiindex.

# Assume the array is like these : for concept..
arrays = [['male', 'male', 'female', 'female'],
          ['young', 'old', 'young', 'old']]

index = pd.MultiIndex.from_arrays(arrays, names=('Gender', 'Type'))
df = pd.DataFrame({'Max Enthu': [390., 350., 30., 20.]},
                  index=index)

print(df)

"""
Name: Data, dtype: int64
              Max Enthu
Gender Type            
male   young      390.0
       old        350.0
female young       30.0
       old         20.0
"""

# using level 0
print(df.groupby(level=0).mean())
"""
        Max Enthu
Gender           
female       25.0
male        370.0

"""

# Using level
print(df.groupby(level="Gender").mean())
"""
       Max Enthu
Type            
old        185.0
young      210.0
"""

print(df.groupby(level="Gender").mean())
"""
       Max Enthu
Gender           
female       25.0
male        370.0
"""

Happy Learning at AMET!!!

Fundas2

Some Basic Functions

# for <loop_variable> in <iterable>:
#     <code>
for i in range(10):
    print(i)
print(25*'-')
# for <loop_variable> in range(<start>, <stop>, <step>):
#     <code>
for i in range(1,10,2):
    print(i)
for i in range(1,30,2):
    print('fun'*i)
for i in range(30,1,-2):
    print('fun'*i)
l = ['apple','boy','cat','dog']
# for loop with iterable list
for i in range(len(l)):
   print(l[i])
str = ' I am iterable by seperation?'
for i in str:
    print(i)


# Break
lis = [1, 2, 3, 4, 5]


for elem in lis:
   if elem % 2 == 0:
      print("Even:", elem)
      print("break")
      break
   else:
      print("Odd:", elem)
"""
Odd: 1
Even: 2
break
"""
# Continue

for elem in lis:
   if elem % 2 == 0:
      print("continue")
      continue
   print("Odd:", elem)
"""
Odd: 1
continue
Odd: 3
continue
Odd: 5
"""

# zip() is an amazing built-in function that we can use in Python
# to iterate over multiple sequences at once,
# getting their corresponding elements in each iteration

list1 = [10, 20, 30, 40]
list2 = [50, 60, 70, 80]
list3 = ['a','b','c','d']

for elem1, elem2,elem3 in zip(list1, list2, list3):
    print(elem1, elem2, elem3)
"""
10 50 a
20 60 b
30 70 c
40 80 d
"""


# while <condition>:
#     <code>

x=5
while x >= 0:
   print("Fun " * x)
   x -= 1

#nested for loop
dim =3
for i in range(dim):
    for j in range(dim):
        for k in range(dim):
            print(i,j,k)
        print('\n')
    print('\n') # 3 x 3 3 = 9 elements x 9

num_cols=5
num_rows=5
for i in range(5):
   for num_cols in range(num_rows-i):
      print("*", end="")
   print()


# Function in pytho

def fun1():
    print('def()')

fun1()

def add(x,y):
    print(x+y)
add(5,5)


def mulp(a, b=5):  # default value for b
    print(a * b)

mulp(10) #50


# Recursibve Factorial function

def fact(n):
    if n == 0 or n == 1:
        return 1
    else:
        return n * fact(n-1)

f = fact(5)
print(f)#120

def fib(n):
    if n == 0 or n == 1:
        return n
    else:
        return fib(n-1) + fib(n-2)
print(5*'-')
f = fib(4)
print(f) #3

Class : Object oriented concepts in python using class

class student:
    name =  'Steve Jobs',
    co = 'Apple',
    country = 'USA',
    sal = 1000000

    def __init__(self,param1, param2, param3, param4):
        self.name = param1
        self.co = param2
        self.country = param3
        self.sal = param4

    def display(self, param1,param2, param3, param4):
        print(param1,param2,param3, param4)


stu = student('Bill Gates','Micro Soft','USA',100000)
print(stu.name, stu.sal) # Bill Gates 100000

print(type(stu)) #<class '__main__.student'>
print(stu) # <__main__.student object at 0x000001ACB0C19FD0>
print(stu.co,stu.country) #Micro Soft USA

# del stu

#---------------------------------------------
class Student:

    def __init__(self, name):
        self._name = name

    @property
    def name(self):
        print("Calling getter")
        return self._name

    @name.setter
    def name(self, new_name):
        print("Calling setter")
        self._name = new_name

    @name.deleter
    def name(self):
        print("Calling deleter")
        del self._name

stu = Student("Sandilya")
print(stu.name)
stu.name = "Chandra"
print(stu.name)

del stu
"""
Calling getter
Sandilya
Calling setter
Calling getter
Chandra Gupta
"""

All about import statement

Many ways we can import a python Libraries we will see the various ways

import pandas
print(pandas.read_csv('bio.csv'))

import pandas as pd  # renamed as pd
df = pd.read_csv('bio.csv')
print(df)

from pandas import read_csv  # import only read_csv
print(read_csv('bio.csv'))

"""
   Unnamed: 0  Name  Age
0           0   Raj   23
1           1   Ram   23
2           2  Sita   24
3           3  Laks   21
"""

from pandas import *  # import all 
print('df:',read_csv('bio.csv'))
"""
df:    Unnamed: 0  Name  Age
0           0   Raj   23
1           1   Ram   23
2           2  Sita   24
3           3  Laks   21
"""

List Comprehension in Python

The syntax used to define list comprehensions usually follows one of these four patterns:

• [<value_to_include> for <var> in <sequence>]
• [<value_to_include> for <var1> in <sequence1> for <var2> in <sequence2>][<value_to_include> for <var> in <sequence> if <condition>]
• [<value> for <var1> in <sequence1> for <var2> in <sequence2> if <condition>]

# [<value_to_include> for <var> in <sequence>]
#print all alphabets
print([chr(i) for i in range(65, 91)])
#['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z']

# [<value_to_include> for <var1> in <sequence1> for <var2> in <sequence2>][<value_to_include> for <var> in <sequence> if <condition>]
print([k for k in range(1, 25) if k % 2 == 0])
#[2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24]
# [<value> for <var1> in <sequence1> for <var2> in <sequence2> if <condition>]
print([i * j for i in range(1, 5) for j in range(1, 5) if i % j == 0])
# [1, 2, 4, 3, 9, 4, 8, 16]

Above sample code snippets will explain how to use list comprehension with for, nested for, and if condition.

# Brain Teaser See the differenc  between the below statements

import sys
print(sys.getsizeof([i for i in range(500)])) # 4216

print(sys.getsizeof((i for i in range(500))))   #112

Here all elements in a list and versus one element in a list at a time. That's why, there is difference in memory allocation.