DATA SCIENCE ETHICS

 DSE definition:

“Data ethics is a new branch of ethics that studies and evaluates moral problems related to data (including generation, recording, curation, processing, dissemination, sharing and use), algorithms (including artificial intelligence, artificial agents, machine learning and robots) and corresponding practices (including responsible innovation, programming, hacking and professional codes), in order to formulate and support morally good solutions (e.g. right conducts or right values).”

MongoDB - Introduction

 MongaDB Features

• Full cloud-based application data platform.
• Flexible document schemas.
• JSON..BSON (BJSON)..
• Widely supported and code-native data access.
• Change-friendly design.
• Powerful querying and analytics.
• Easy horizontal scale-out with sharding.
• Simple installation.
• Cost-effective.

Learning to use MongoDB

Step 1:

Download and install "PyMongo":

                pip install mongadb

Step 2:

                    import pymonga

step 3: list database names

                # import library

import pymongo
# creating a client
myc = pymongo.MongoClient("mongodb://localhost:27017/")
# Just printing existing databases
print(myc.list_database_names()) # ['admin', 'config', 'db1', 'local']

Step 4: Create Collections

# import library

import pymongo
# creating a client
c = pymongo.MongoClient("mongodb://localhost:27017/")
# Just printing existing databases
print(c.list_database_names()) # ['admin', 'config', 'db1', 'local']

# choose any demo db as db1
# create collection customers in db1
db = c['db1']
col = db["customers"]
print(db.list_collection_names())

# add Dict data
dict = { "name": "John", "address": "Highway 37" }
x = col.insert_one(dict)
print(x.inserted_id)

#add Dict1 data
dict2 = { "name": "Peter", "address": "Lowstreet 27" }
y = col.insert_one(dict2)
print(y.inserted_id)

"""
['admin', 'config', 'db1', 'local']
['customers']
62bd57445c7e7af908e87efa
62bd57445c7e7af908e87efb

"""

To display with Object_ID

clist = db.list_collection_names()
if "customers" in clist:
  print("The collection exists.")

for x in col.find():
   print(x)

"""
['admin', 'config', 'local']
[]
62bd58d899982b27053eb9cc
62bd58d899982b27053eb9cd
The collection exists.
{'_id': ObjectId('62bd58d899982b27053eb9cc'), 'name': 'John', 'address': 'Highway 37'}
{'_id': ObjectId('62bd58d899982b27053eb9cd'), 'name': 'Peter', 'address': 'Lowstreet 27'}
"""

Before and After deleting 

# import library

import pymongo
# creating a client
c = pymongo.MongoClient("mongodb://localhost:27017/")
db = c['db1']
col = db["customers"]
col.drop()
q1 = { 'address': 'Highway 37'}
q2 = { 'address': 'Highway 38'}
q3 = { 'address': 'Highway 39'}

print(10*'-', 'Insering 3 qs')
col.insert_one(q1)
col.insert_one(q2)
col.insert_one(q3)

print(10*'-', 'Before Delete')
for x in col.find():
   print(x)

q4 = { 'address': 'Highway 39'}
col.delete_many(q4)
print(10*'-', 'After Delete')
for x in col.find():
   print(x)
"""
---------- Insering 3 qs
---------- Before Delete
{'_id': ObjectId('62bd5db330e379663c8851ce'), 'address': 'Highway 37'}
{'_id': ObjectId('62bd5db430e379663c8851cf'), 'address': 'Highway 38'}
{'_id': ObjectId('62bd5db430e379663c8851d0'), 'address': 'Highway 39'}
---------- After Delete
{'_id': ObjectId('62bd5db330e379663c8851ce'), 'address': 'Highway 37'}
{'_id': ObjectId('62bd5db430e379663c8851cf'), 'address': 'Highway 38'}
"""

After deleting another Record

print('After Prinitng')

q5 = { 'address': 'Highway 37'}
col.delete_many(q5)
for x in col.find():
   print(x)
"""
After Prinitng
{'_id': ObjectId('62bd5f3cf32b68f5a744b5ab'), 'address': 'Highway 38'}\
"""

Before and After dropping collections

import pymongo
# creating a client
c = pymongo.MongoClient("mongodb://localhost:27017/")
db = c['db1']
col = db["customers"]

many = [
  { "_id": 1, "name": "AMET", "address": "ECR,  Chennai"},
  { "_id": 2, "name": "ABS", "address": "Anna Nagar, Chennai"},
]
x = col.insert_many(many)

print('Before dropping')
for x in col.find():
       print(x)
col.drop()
print('After dropping')
for x in col.find():
       print(x)
"""
Before dropping
{'_id': 1, 'name': 'AMET', 'address': 'ECR,  Chennai'}
{'_id': 2, 'name': 'ABS', 'address': 'Anna Nagar, Chennai'}
After dropping

"""

Sorting Collections

import pymongo

# creating a client
c = pymongo.MongoClient("mongodb://localhost:27017/")
db = c['db1']
col = db["customers"]
col.drop()
many = [
  { "_id": 11, "name": "AMET", "address": "ECR,  Chennai"},
  { "_id": 21, "name": "ABS", "address": "Anna Nagar, Chennai"},
]
x = col.insert_many(many)
print('Before Sorting')
for x in col.find():
       print(x)
doc = col.find().sort("name")

print('After Sorting')
for x in doc:
  print(x)
  
"""
Before Sorting
{'_id': 11, 'name': 'AMET', 'address': 'ECR,  Chennai'}
{'_id': 21, 'name': 'ABS', 'address': 'Anna Nagar, Chennai'}
After Sorting
{'_id': 21, 'name': 'ABS', 'address': 'Anna Nagar, Chennai'}
{'_id': 11, 'name': 'AMET', 'address': 'ECR,  Chennai'}
"""

Before and After Updating

import pymongo

# creating a client
c = pymongo.MongoClient("mongodb://localhost:27017/")
db = c['db1']
col = db["customers"]
col.drop()
many = [
  { "_id": 11, "name": "AMET", "address": "ECR,  Chennai"},
  { "_id": 21, "name": "ABS", "address": "Anna Nagar, Chennai"},
]
x = col.insert_many(many)
q6 = { "address": 'ECR,  Chennai' }
newvalues = { "$set": { "address": "#123, East Coast Road,  Chennai" } }
print('Before Updation')
for x in col.find():
  print(x)
col.update_one(q6, newvalues)
print("After Updation")
#print "customers" after the update:
for x in col.find():
  print(x)
"""
Before Updation
{'_id': 11, 'name': 'AMET', 'address': 'ECR,  Chennai'}
{'_id': 21, 'name': 'ABS', 'address': 'Anna Nagar, Chennai'}
After Updation
{'_id': 11, 'name': 'AMET', 'address': '#123, East Coast Road,  Chennai'}
{'_id': 21, 'name': 'ABS', 'address': 'Anna Nagar, Chennai'}
"""

B4 and After Many Updating using regular expression

import pymongo

client = pymongo.MongoClient("mongodb://localhost:27017/")
db = client["mydatabase"]
col = db["customers"]
col.drop()

many = [
  { "_id": 10, "name": "AMET", "address": "ECR,  Chennai"},
  { "_id": 12, "name": "ABS", "address": "Anna Nagar, Chennai"},
  { "_id": 13, "name": "ABS!@", "address": "ECR,  Chennai"},
  { "_id": 14, "name": "Stanford", "address": "Anna Nagar, Chennai"},
]
x = col.insert_many(many)


print('Before Updation')
for x in col.find():
   print(x)

query = { "address": { "$regex": "^A" } }
newvalues = { "$set": { "name": "AMET Updated" } }
x = col.update_many(query, newvalues)
print(x.modified_count, "documents updated.")

print('After many Updation')
for x in col.find():
   print(x)
"""
Before Updation
{'_id': 10, 'name': 'AMET', 'address': 'ECR,  Chennai'}
{'_id': 12, 'name': 'ABS', 'address': 'Anna Nagar, Chennai'}
{'_id': 13, 'name': 'ABS!@', 'address': 'ECR,  Chennai'}
{'_id': 14, 'name': 'Stanford', 'address': 'Anna Nagar, Chennai'}
2 documents updated.
After many Updation
{'_id': 10, 'name': 'AMET', 'address': 'ECR,  Chennai'}
{'_id': 12, 'name': 'AMET Updated', 'address': 'Anna Nagar, Chennai'}
{'_id': 13, 'name': 'ABS!@', 'address': 'ECR,  Chennai'}
{'_id': 14, 'name': 'AMET Updated', 'address': 'Anna Nagar, Chennai'}
"""

After using limit to display 

# limit parameter to limit two records
output = col.find().limit(2)
print('After setting Limit(2)')
for x in output:
    print(x)
"""
After setting Limit(2)
{'_id': 10, 'name': 'AMET', 'address': 'ECR,  Chennai'}
{'_id': 12, 'name': 'AMET Updated', 'address': 'Anna Nagar, Chennai'}
"""

Bon Appetite and Happy MongoDB Coding....Learning!!!

Recommendations System - Course for +2 Students

 Recommendation System using pandas and Sklearn

Student Dataset1.csv

id,gender,stream,subject,marks,course,specialization
1,m,science,physics,78,btech,civil
2,f,commerce,maths,56,bsc,maths
3,m,humanities,history,79,ba,history
4,f,commerce,economics,88,bcom,hons
5,m,science,chemistry,79,bsc,biology
6,m,science,maths,98,btech,mech
7,f,commerce,physics,56,bsc,maths
8,m,humanities,history,79,ba,history
9,f,commerce,commerce,76,bcom,hons
10,m,science,physics,89,bsc,biology

Code snippet (Thanks to  Siddharth Sachdeva — August 25, 2021, Analytics Vidya)

#S1 importing necessary libraries.
import pandas as pd
data = pd.read_csv(r"dataset1.csv")
#s2
descriptions =data['gender'] +' '+ data['subject'] + ' ' + data['stream'] +' '+ data['marks'].apply(str)
#Printing the first description
print(descriptions[0])
# s3 similarity matrix
#import sklearn.TfidfVector
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import linear_kernel
def create_similarity_matrix(new_description, overall_descriptions):
        overall_descriptions.append(new_description)
        # Define a tfidf [Term Frequency Inverse document frequency]  vectorizer and remove all stopwords.
        tfidf = TfidfVectorizer(stop_words="english")
        tfidf_matrix = tfidf.fit_transform(overall_descriptions)
        tfidf_matrix.shape
        cosine_sim = linear_kernel(tfidf_matrix,tfidf_matrix)
        return cosine_sim
#s5 Sim Matrix reco
def get_recommendations(new_description,overall_descriptions):

        cosine_sim = create_similarity_matrix(new_description,overall_descriptions)
        sim_scores = list(enumerate(cosine_sim[-1]))
        sim_scores = sorted(sim_scores,key =lambda x:x[1],reverse= True )
        sim_scores = sim_scores[1:10]
        indices = [i[0]for i in sim_scores]
        return data.iloc[indices]
# s6 Recommendations
new_description = pd.Series('m physics science 78')
recommendation= get_recommendations(new_description,descriptions)
print(recommendation[0:3])
"""
   id gender    stream    subject  marks course specialization
0   1      m   science    physics     78  btech          civil
6   7      f  commerce    physics     56    bsc          maths
4   5      m   science  chemistry     79    bsc        biology
"""

Simple not refined. But throw some light on basics of Recommendation System.

P#32 Reg expression

To find a pattern in a statement

match = re.search('love', 'i love you all')

if match:
    print('found love')
else:
    print('love Not found')

# found love

match = re.search('love', 'i leave you all')

if match:
    print('found love')
else:
    print('love Not found')

# love Not found

 ## . = any char but \n
match = re.search(r'..ve', 'i love you all') # found,
print(match)

To match email

 to check up Google Account
str = 'hello I am so& so i have account you@google.com'
match = re.search(r'\w+@\w+', str)
if match:
    print(match.group()) #you@google

match = re.search(r'[\w.-]+@[\w.-]+', str)
if match:
    print(match.group())  ## 'you@google.com'

To Validate IFSC code in banks

import re
def isValidIFSCode(str):
    # Regex to check valid IFSC Code.
    regex = "^[A-Z]{4}0[A-Z0-9]{6}$"  # 4 Alphabets,0, 
    # Compile the ReGex
    p = re.compile(regex)
    # If the string is empty
    # return false
    if (str == None):
        return False
    # Return if the string
    # matched the ReGex
    if (re.search(p, str)):
        return True
    else:
        return False
# Driver code
# Test Case 1:
str1 = "SBIN0125620"
print(isValidIFSCode(str1))

# Test Case 2:
str2 = "SBIN0125"
print(isValidIFSCode(str2))

# Test Case 3:
str3 = "1234SBIN012"
print(isValidIFSCode(str3))

# Test Case 4:
str4 = "SBIN7125620"
print(isValidIFSCode(str4))

# This code is contributed by avanitrachhadiya2155

"""
True
False
False
False
"""

Please Note:

# regex = "^[A-Z]{4}0[A-Z0-9]{6}$";
# Where: 
# ^ represents the starting of the string.
# [A-Z]{4} represents the first four characters should be upper case alphabets.
# 0 represents the fifth character should be 0.
# [A-Z0-9]{6} represents the next six characters usually numeric, but can also be alphabetic.
# $ represents the ending of the string.

To test Pin Code

import re

def isValidPinCode(pinCode):
    # Regex to check valid pin code
    # of India.
    regex = "^[1-9]{1}[0-9]{2}\\s{0,1}[0-9]{3}$";

    # Compile the ReGex
    p = re.compile(regex);

    # If the pin code is empty
    # return false
    if (pinCode == ''):
        return False;

    # Pattern class contains matcher() method
    # to find matching between given pin code
    # and regular expression.
    m = re.match(p, pinCode);

    # Return True if the pin code
    # matched the ReGex else False
    if m is None:
        return False
    else:
        return True



# check Driver code
if __name__ == "__main__":
    # Test case 1
    num1 = "625001";
    print(num1, ": ", isValidPinCode(num1));

    # Test case 2:
    num2 = "625 001";
    print(num2, ": ", isValidPinCode(num2));

    # Test case 3:
    num3 = "0625 001";
    print(num3, ": ", isValidPinCode(num3));

    # Test case 4:
    num4 = "6250001";
    print(num4, ": ", isValidPinCode(num4));

Result

"""
625001 :  True
625 001 :  True
0625 001 :  False
6250001 :  False
"""

To check PANCARD in India

# To Valiate pan Card

import re

def checkPanCard():
        import re


        def isValidPanCardNo(panCardNo):

            # PAN Card number
            regex = "[A-Z]{5}[0-9]{4}[A-Z]{1}"

            # Compile the ReGex
            p = re.compile(regex)

            # If the PAN Card number
            # is empty return false
            if(panCardNo == None):
                return False

            # Return if the PAN Card number
            # matched the ReGex
            if(re.search(p, panCardNo) and
            len(panCardNo) == 10):
                return True
            else:
                return False

        # Driver Code.


        # Test Case 1:
        str1 = "AFGHA2318J"
        print(isValidPanCardNo(str1),' ',str1)

        # Test Case 2:
        str2 = "23AFGHN18J"
        print(isValidPanCardNo(str2),str2)

        # Test Case 3:
        str3 = "AFGHA2318JM"
        print(isValidPanCardNo(str3),str3)

        # Test Case 4:
        str4 = "AFGHA23184"
        print(isValidPanCardNo(str4),str4)

        # Test Case 5:
        str5 = "AFGHA 23184"
        print(isValidPanCardNo(str5),str5)

checkPanCard()
"""
True   AFGHA2318J
False 23AFGHN18J
False AFGHA2318JM
False AFGHA23184
False AFGHA 23184
"""

P#31 Magic Methods

 Magic or Dunder Methods

Methods having two prefix and suffix underscores in the method name. 

Dunder here means “Double Under (Underscores)”

Sample Method:

# declare our test string class
class Str:

    # magic method to initiate object
    def __init__(self, string):
        self.string = string


# Driver Code
if __name__ == '__main__':
    # object creation
    s1 = Str('Hello Python')

    # print object location
    print(s1)

To print __ methods of int

print(dir(int))

# Result

"""
['__abs__', '__add__', '__and__', '__bool__', '__ceil__', '__class__',
 '__delattr__', '__dir__', '__divmod__', '__doc__', '__eq__', '__float__', 
 '__floor__', '__floordiv__', '__format__', '__ge__', '__getattribute__', 
 '__getnewargs__', '__gt__', '__hash__', '__index__', '__init__', '__init_subclass__',
  '__int__', '__invert__', '__le__', '__lshift__', '__lt__', '__mod__', '__mul__', 
  '__ne__', '__neg__', '__new__', '__or__', '__pos__', '__pow__', '__radd__', '__rand__',
  '__rdivmod__', '__reduce__', '__reduce_ex__', '__repr__', '__rfloordiv__', '__rlshift__',
  '__rmod__', '__rmul__', '__ror__', '__round__', '__rpow__', '__rrshift__', '__rshift__', '__rsub__', '__rtruediv__',
  '__rxor__', '__setattr__', '__sizeof__', '__str__', '__sub__', '__subclasshook__', '__truediv__',
  '__trunc__', '__xor__', 'as_integer_ratio', 'bit_length', 'conjugate', 'denominator', 'from_bytes', 'imag', 
  'numerator', 'real', 'to_bytes']
"""

Important Magic Methods

The following tables list important magic methods in Python 3.

Initialization and Construction	Description
__new__(cls, other)	To get called in an object's instantiation.
__init__(self, other)	To get called by the __new__ method.
__del__(self)	Destructor method.

Unary operators and functions	Description
__pos__(self)	To get called for unary positive e.g. +someobject.
__neg__(self)	To get called for unary negative e.g. -someobject.
__abs__(self)	To get called by built-in abs() function.
__invert__(self)	To get called for inversion using the ~ operator.
__round__(self,n)	To get called by built-in round() function.
__floor__(self)	To get called by built-in math.floor() function.
__ceil__(self)	To get called by built-in math.ceil() function.
__trunc__(self)	To get called by built-in math.trunc() function.

Augmented Assignment	Description
__iadd__(self, other)	To get called on addition with assignment e.g. a +=b.
__isub__(self, other)	To get called on subtraction with assignment e.g. a -=b.
__imul__(self, other)	To get called on multiplication with assignment e.g. a *=b.
__ifloordiv__(self, other)	To get called on integer division with assignment e.g. a //=b.
__idiv__(self, other)	To get called on division with assignment e.g. a /=b.
__itruediv__(self, other)	To get called on true division with assignment
__imod__(self, other)	To get called on modulo with assignment e.g. a%=b.
__ipow__(self, other)	To get called on exponentswith assignment e.g. a **=b.
__ilshift__(self, other)	To get called on left bitwise shift with assignment e.g. a<<=b.
__irshift__(self, other)	To get called on right bitwise shift with assignment e.g. a >>=b.
__iand__(self, other)	To get called on bitwise AND with assignment e.g. a&=b.
__ior__(self, other)	To get called on bitwise OR with assignment e.g. a|=b.
__ixor__(self, other)	To get called on bitwise XOR with assignment e.g. a ^=b.

Type Conversion Magic Methods	Description
__int__(self)	To get called by built-int int() method to convert a type to an int.
__float__(self)	To get called by built-int float() method to convert a type to float.
__complex__(self)	To get called by built-int complex() method to convert a type to complex.
__oct__(self)	To get called by built-int oct() method to convert a type to octal.
__hex__(self)	To get called by built-int hex() method to convert a type to hexadecimal.
__index__(self)	To get called on type conversion to an int when the object is used in a slice expression.
__trunc__(self)	To get called from math.trunc() method.

String Magic Methods	Description
__str__(self)	To get called by built-int str() method to return a string representation of a type.
__repr__(self)	To get called by built-int repr() method to return a machine readable representation of a type.
__unicode__(self)	To get called by built-int unicode() method to return an unicode string of a type.
__format__(self, formatstr)	To get called by built-int string.format() method to return a new style of string.
__hash__(self)	To get called by built-int hash() method to return an integer.
__nonzero__(self)	To get called by built-int bool() method to return True or False.
__dir__(self)	To get called by built-int dir() method to return a list of attributes of a class.
__sizeof__(self)	To get called by built-int sys.getsizeof() method to return the size of an object.

Attribute Magic Methods	Description
__getattr__(self, name)	Is called when the accessing attribute of a class that does not exist.
__setattr__(self, name, value)	Is called when assigning a value to the attribute of a class.
__delattr__(self, name)	Is called when deleting an attribute of a class.

Operator Magic Methods	Description
__add__(self, other)	To get called on add operation using + operator
__sub__(self, other)	To get called on subtraction operation using - operator.
__mul__(self, other)	To get called on multiplication operation using * operator.
__floordiv__(self, other)	To get called on floor division operation using // operator.
__truediv__(self, other)	To get called on division operation using / operator.
__mod__(self, other)	To get called on modulo operation using % operator.
__pow__(self, other[, modulo])	To get called on calculating the power using ** operator.
__lt__(self, other)	To get called on comparison using < operator.
__le__(self, other)	To get called on comparison using <= operator.
__eq__(self, other)	To get called on comparison using == operator.
__ne__(self, other)	To get called on comparison using != operator.
__ge__(self, other)	To get called on comparison using >= operator.