P#30 H/W Internals and Profiling

To know System Information through Python

# Know system Information
# pip install wmi

import wmi

computer = wmi.WMI()
our_system = computer.Win32_ComputerSystem()[0]

print(f"Manufacturer: {our_system.Manufacturer}")
print(f"Model: {our_system. Model}")
print(f"Name: {our_system.Name}")
print(f"NumberOfProcessors: {our_system.NumberOfProcessors}")
print(f"SystemType: {our_system.SystemType}")
print(f"SystemFamily: {our_system.SystemFamily}")
""" Output from my system:
Manufacturer: Dell Inc.

Model: Vostro 3888

Name: MECH1

NumberOfProcessors: 1

SystemType: x64-based PC

SystemFamily: Vostro
"""

Simple System info Python program using platform

# importing module
import platform

# dictionary
info = {}

# platform details
platform_details = platform.platform()

# adding it to dictionary
info["platform details"] = platform_details

# system name
system_name = platform.system()

# adding it to dictionary
info["system name"] = system_name

# processor name
processor_name = platform.processor()

# adding it to dictionary
info["processor name"] = processor_name

# architectural detail
architecture_details = platform.architecture()

# adding it to dictionary
info["architectural detail"] = architecture_details

# printing the details
for i, j in info.items():
   print(i, " - ", j)

"""
platform details  -  Windows-10-10.0.19043-SP0
system name  -  Windows
processor name  -  Intel64 Family 6 Model 165 Stepping 3, GenuineIntel
architectural detail  -  ('64bit', 'WindowsPE')

Process finished with exit code 0
"""

Hardware Information using PSUTIL

import psutil

#Physical cores
print(f"Number of physical cores: {psutil.cpu_count(logical=False)}")
#Logical cores
print(f"Number of logical cores: {psutil.cpu_count(logical=True)}")

#Current frequency
print(f"Current CPU frequency: {psutil.cpu_freq().current}")
#Min frequency
print(f"Min CPU frequency: {psutil.cpu_freq().min}")
#Max frequency
print(f"Max CPU frequency: {psutil.cpu_freq().max}")
# cpu util
#System-wide CPU utilization
print(f"Current CPU utilization: {psutil.cpu_percent(interval=1)}")
#System-wide per-CPU utilization
print(f"Current per-CPU utilization: {psutil.cpu_percent(interval=1, percpu=True)}")

#ram details
#Total RAM
print(f"Total RAM installed: {round(psutil.virtual_memory().total/1000000000, 2)} GB")
#Available RAM
print(f"Available RAM: {round(psutil.virtual_memory().available/1000000000, 2)} GB")
#Used RAM
print(f"Used RAM: {round(psutil.virtual_memory().used/1000000000, 2)} GB")
#RAM usage
print(f"RAM usage: {psutil.virtual_memory().percent}%")

Results

"""
Number of physical cores: 4
Number of logical cores: 8
Current CPU frequency: 3600.0
Min CPU frequency: 0.0
Max CPU frequency: 3600.0
Current CPU utilization: 4.2
Current per-CPU utilization: [15.7, 0.0, 25.0, 0.0, 4.5, 0.0, 1.5, 1.6]
Total RAM installed: 8.33 GB
Available RAM: 1.58 GB
Used RAM: 6.75 GB
RAM usage: 81.0%
"""

Disk and N/W Information using Python

# Disk Usage

import psutil
import platform
from datetime import datetime

def get_size(bytes, suffix="B"):
    """
    Scale bytes to its proper format
    e.g:
        1253656 => '1.20MB'
        1253656678 => '1.17GB'
    """
    factor = 1024
    for unit in ["", "K", "M", "G", "T", "P"]:
        if bytes < factor:
            return f"{bytes:.2f}{unit}{suffix}"
        bytes /= factor

# Disk Information
print("="*40, "Disk Information", "="*40)
print("Partitions and Usage:")
# get all disk partitions
partitions = psutil.disk_partitions()
for partition in partitions:
    print(f"=== Device: {partition.device} ===")
    print(f"  Mountpoint: {partition.mountpoint}")
    print(f"  File system type: {partition.fstype}")
    try:
        partition_usage = psutil.disk_usage(partition.mountpoint)
    except PermissionError:
        # this can be catched due to the disk that
        # isn't ready
        continue
    print(f"  Total Size: {get_size(partition_usage.total)}")
    print(f"  Used: {get_size(partition_usage.used)}")
    print(f"  Free: {get_size(partition_usage.free)}")
    print(f"  Percentage: {partition_usage.percent}%")
# get IO statistics since boot
disk_io = psutil.disk_io_counters()
print(f"Total read: {get_size(disk_io.read_bytes)}")
print(f"Total write: {get_size(disk_io.write_bytes)}")

# N/w
# Network information
print("="*40, "Network Information", "="*40)
# get all network interfaces (virtual and physical)
if_addrs = psutil.net_if_addrs()
for interface_name, interface_addresses in if_addrs.items():
    for address in interface_addresses:
        print(f"=== Interface: {interface_name} ===")
        if str(address.family) == 'AddressFamily.AF_INET':
            print(f"  IP Address: {address.address}")
            print(f"  Netmask: {address.netmask}")
            print(f"  Broadcast IP: {address.broadcast}")
        elif str(address.family) == 'AddressFamily.AF_PACKET':
            print(f"  MAC Address: {address.address}")
            print(f"  Netmask: {address.netmask}")
            print(f"  Broadcast MAC: {address.broadcast}")
# get IO statistics since boot
net_io = psutil.net_io_counters()
print(f"Total Bytes Sent: {get_size(net_io.bytes_sent)}")
print(f"Total Bytes Received: {get_size(net_io.bytes_recv)}")

Result:

"""
======================================== Disk Information ========================================
Partitions and Usage:
=== Device: C:\ ===
  Mountpoint: C:\
  File system type: NTFS
  Total Size: 194.70GB
  Used: 111.24GB
  Free: 83.46GB
  Percentage: 57.1%
=== Device: D:\ ===
  Mountpoint: D:\
  File system type: NTFS
  Total Size: 736.20GB
  Used: 4.03GB
  Free: 732.16GB
  Percentage: 0.5%
Total read: 61.63GB
Total write: 46.68GB
======================================== Network Information ========================================
=== Interface: Local Area Connection* 11 ===
=== Interface: Local Area Connection* 11 ===
  IP Address: 169.254.62.94
  Netmask: 255.255.0.0
  Broadcast IP: None
=== Interface: Local Area Connection* 11 ===
=== Interface: Local Area Connection* 12 ===
=== Interface: Local Area Connection* 12 ===
  IP Address: 169.254.72.255
  Netmask: 255.255.0.0
  Broadcast IP: None
=== Interface: Local Area Connection* 12 ===
=== Interface: Ethernet ===
=== Interface: Ethernet ===
  IP Address: 172.20.24.131
  Netmask: 255.255.0.0
  Broadcast IP: None
=== Interface: Ethernet ===
=== Interface: Wi-Fi 2 ===
=== Interface: Wi-Fi 2 ===
  IP Address: 172.20.29.139
  Netmask: 255.255.0.0
  Broadcast IP: None
=== Interface: Wi-Fi 2 ===
=== Interface: Loopback Pseudo-Interface 1 ===
  IP Address: 127.0.0.1
  Netmask: 255.0.0.0
  Broadcast IP: None
=== Interface: Loopback Pseudo-Interface 1 ===
Total Bytes Sent: 366.06MB
Total Bytes Received: 4.38GB
"""

Measuring Running Time in Python  #1

import timeit
time_test = timeit.repeat('[x**0.5 for x in range(1000)]', number=10000)
print(time_test)

Result:

"""
[1.415492, 1.3847563, 1.3762589999999997, 1.3683869, 1.3789484000000005]
"""

# importing cProfile
import cProfile

cProfile.run("10 + 10")

"""
   3 function calls in 0.000 seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    0.000    0.000 <string>:1(<module>)
        1    0.000    0.000    0.000    0.000 {built-in method builtins.exec}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
"""

Function Profiling code and the corresponding output

# Function Profiling
import cProfile

def f():
   print("Hello Python World Profiling")
cProfile.run('f()')

         5 function calls in 0.000 seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    0.000    0.000 <string>:1(<module>)
        1    0.000    0.000    0.000    0.000 ex1-scit.py:252(f)
        1    0.000    0.000    0.000    0.000 {built-in method builtins.exec}
        1    0.000    0.000    0.000    0.000 {built-in method builtins.print}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

LAB WORK

 Lab Work:

1.  Python Environment setup and Essentials. 

2.  Mathematical computing with Python (NumPy).

3.  Scientific Computing with Python (SciPy).  Numerical & Scientific Python Notes

4.  Data Manipulation with Pandas.

5.  Prediction using Scikit-Learn

6.  Data Visualization in python using matplotlib

Inspiring Industry Visualization Projects

 VISUALISATION

What you imagine is what you see on board/screen.

Information is beautiful. Knowledge is beautiful.

=  information & function & visual form &  story (IFVS)

Refer 1 here 

Refer 2 here

Refer for Data Visualization Good Practices(5)

Best Visualization Samples

Best Dashboard

Anatomy of a Figure

Machine Learning with Matplot

Play with Pandas [Data manipulation with Data]

Case Studies 

MDS

Marine data Science 

LDS - Logistics Data Science

Scipy Intro Tutorial with answers

 Tutorial on scipy intro..

# Tutorial on Scipy intro
# SciPy is a scientific computation library above NumPy.
# SciPy stands for Scientific Python.
# Ref w3schools.com


from scipy import constants
print(constants.liter) # 0.001
print(constants.pi) # 3.141592653589793


import scipy
print(scipy.__version__) # 1.8.1


from scipy import constants
print(constants.yotta)    #1e+24
print(constants.zetta)    #1e+21
print(constants.exa)      #1e+18
print(constants.peta)     #1000000000000000.0
print(constants.tera)     #1000000000000.0
print(constants.giga)     #1000000000.0
print(constants.mega)     #1000000.0
print(constants.kilo)     #1000.0
print(constants.hecto)    #100.0
print(constants.deka)     #10.0
print(constants.deci)     #0.1
print(constants.centi)    #0.01
print(constants.milli)    #0.001
print(constants.micro)    #1e-06
print(constants.nano)     #1e-09
print(constants.pico)     #1e-12
print(constants.femto)    #1e-15
print(constants.atto)     #1e-18
print(constants.zepto)    #1e-21
#
# # Root of an equation ..optimizer
from scipy.optimize import root
from math import cos

def eqn(x):
  return x + cos(x)
myroot = root(eqn, 0)
print(myroot.x)         #[-0.73908513]

#Sparse Matrix to avoid unused elements
import numpy as np
from scipy.sparse import csr_matrix
arr = np.array([0, 0, 0, 0, 0, 1, 1, 0, 2])
print(csr_matrix(arr))
'''
  (0, 5)   1
  (0, 6)   1
  (0, 8)   2
'''

import numpy as np
from scipy.sparse import csr_matrix

arr = np.array([[0, 0, 0], [0, 0, 1], [1, 0, 2]])

mat = csr_matrix(arr)
mat.eliminate_zeros()

print(mat)
'''
 (0, 5)    1
  (0, 6)   1
  (0, 8)   2
  (1, 2)   1
  (2, 0)   1
  (2, 2)   2
'''


#SCIPY sIGNIFICANt aspects

import numpy as np
from scipy.stats import ttest_ind

x1 = np.random.normal(size=50)
x2 = np.random.normal(size=50)
#T-Test
print( ttest_ind(x1, x2))
 #Ttest_indResult(statistic=-0.4602568443314745, pvalue=0.6463514741682714)

# pvalue printing
print(ttest_ind(x1, x2).pvalue) #0.3861625359272315


import numpy as np
from scipy.stats import kstest
x = np.random.normal(size=50)
print(kstest(x, 'norm'))
#KstestResult(statistic=0.07440265161334647, pvalue=0.9254263997475219)


#Statistical Distribuition

import numpy as np
from scipy.stats import describe
x = np.random.normal(size=50)
print(describe(x))
# DescribeResult(nobs=50, minmax=(-2.252010033738992, 1.694245094326367),
# mean=-0.34129189546033056, variance=0.7245922182755077, skewness=0.4445284516357619,
# kurtosis=0.11748436830854025)

# Normality Tests (Skewness and Kurtosis)
'''
Skewness:
A measure of symmetry in data, For normal distributions it is 0.
If it is negative, it means the data is skewed left.
If it is positive it means the data is skewed right.
'''
import numpy as np
from scipy.stats import skew, kurtosis

x = np.random.normal(size=50)

print(skew(x))            #0.25667677411004536   skewed right
print(skew(x))            #-0.49188030505407804  skewed left

'''
Kurtosis:
A measure of whether the data is heavy or lightly tailed with 
respect  to a normal distribution.
Positive kurtosis : heavily tailed.
Negative kurtosis : lightly tailed.
'''
print(kurtosis(x))        #-0.37504350194077585 lightly tailed
print(kurtosis(x))        #0.010624259343263276  heavily tailed

from scipy.stats import normaltest
# To check normal distribution of data
print(normaltest(x)) #NormaltestResult(statistic=7.187162674258679,
# pvalue=0.027499668274458256)

Lab Exercises for MBA DS #2

 Let us how Visualization helps in Exploratory Data Analysis or 
An EDA Case Study

Lab exercise for Course work

1.  Python Environment setup and Essentials.
2.  Mathematical computing with Python (NumPy).
3.  Scientific Computing with Python (SciPy).
4.  Data Manipulation with Pandas.
5.  Prediction using Scikit-Learn
6.  Data Visualization in python using matplotlib

Q&A

Difference among between data Analysis, data Analytics and Data Science

Data Analytics	Data Analysis	Data Science
Super set Process of Data Analysis	Sub Set Process of Data Analytics	Set Process
Past and Present study and future e prediction	Past discovery
What will happen in future	What happened
Tools: Tableau, Zoho Analytics,Talend, Hadoop, Xplenty, Kafka,Python,R-Language, Cassandra, MongoDB, HPCC, Spark, Datawrapper, PowerBI etc.	Python, R, PowerBI, Tableau, Excel
Data analytics predicts ‘what will happen next or what is going to be next?’	Data analysis is actually studying past data to understand ‘what happened?’
All analysis and prediction. Data analytics life cycle consists of Business Case Evaluation, Data Identification, Data Acquisition & Filtering, Data Extraction, Data Validation & Cleansing, Data Aggregation & Representation, Data Analysis, Data Visualization, and Utilization of Analysis Results	DA involves Querying, wrangling, statistical modelling, analysis and visualization.	DS involves data sourcing, cleansing, modelling, result evaluation and result testing and deployment

Data Analytics and data Science

Data Science is a combination of multiple disciplines – Mathematics, Statistics, Computer Science, Information Science, Machine Learning, and Artificial Intelligence.

2. Difference between mutable and immutable objects in python

In Python, Built-in projects like int, float, bool, string, tuple are immutable.

In Python, Built-in projects like list, dict, set are mutable.