# Titanic Visions

A gentle introduction to using and extending the visions library.

import pandas as pd
import numpy as np

import visions as v
from visions.typesets import CompleteSet
from visions.functional import infer_type, cast_to_inferred, compare_detect_inference_frame
%matplotlib inline


# The Basics

visions is a library for deterministic evaluation of semantic data types. Under normal conditions, particularly when working with unprocessed data, the type of a series is defined by it’s physical representation in memory, so although it’s trivially obvious to a practitioner that the sequence [1.0, 2.0, 3.0] should be integer, to the machine those are all floats. Here we will demonstrate how you can use visions to easily infer the semantic representations of your data for this and other more complicated data types.

# Load dataset
"https://raw.githubusercontent.com/datasciencedojo/datasets/master/titanic.csv"
)


.dataframe tbody tr th {
vertical-align: top;
}

text-align: right;
}


# Plotting

If you don’t already have gcc compiled you might need to install pygraphviz directly. Visions relies upon pygraphviz to visualize the relationship graph constructed between types.

Let’s make sure pygraphviz is installed so that

!conda install pygraphviz -y

Collecting package metadata (repodata.json): done
Solving environment: done

==> WARNING: A newer version of conda exists. <==
current version: 4.8.1

\$ conda update -n base -c defaults conda

# All requested packages already installed.


# visions Building Blocks

Within visions there are two main ideas we need to get started.

### Types

The first is, unsurprisingly, a Type. Think of these as literally the semantic meaning of a sequence. A String type represents the type for all sequences of exclusively strings (and the missing value indicator np.nan), a DateTime type the type for all sequences of Timestamps or Datetimes, and so on.

### Typesets

The second is the Typeset, which represents a collection of types. Now that we’ve moved beyond the realm of physical types it’s possible to imagine conflicting notions of a sequences data type. For example, some users might be interested in probabilities defined as any sequence of numbers between 0 and 1 while others might instead consider them bound between 0 and 100. In order to resolve this conflict, users can encapsulate their preferred types into a single object, the Typeset. So long as each type follows a few simple rules, visions will automatically determine the relationships between types to construct a traversible relationship graph between each.

typeset = CompleteSet()
typeset.plot_graph()


# Using a Typeset

Now that we have the basic building blocks down, let’s start using our typeset. Our first task is to see visions type inference capabilities in action. In order to make the problem a little more interesting let’s also convert all of our data to strings and see what visions does.

inferred_types = compare_detect_inference_frame(df.astype(str), typeset)
pd.DataFrame({name: [data[i] for data in inferred_types]
for i, name in enumerate(['Columns', 'OriginalType', 'Inferredtype'])})

.dataframe tbody tr th {
vertical-align: top;
}

text-align: right;
}


Great, as we can see, visions is able to exploit the relationships instrinsic to a Typeset to infer the semantic types of our data regardless of it’s initial physical representation.

We can go a little further though and have visions losslessly coerce our data into it’s most semantically meaningful representation without our lifting a finger.

cast_df = cast_to_inferred(df, typeset)

.dataframe tbody tr th {
vertical-align: top;
}

text-align: right;
}


cast_df.dtypes

PassengerId      int64
Survived         int64
Pclass           int64
Name            object
Sex             object
Age            float64
SibSp            int64
Parch            int64
Ticket          object
Fare           float64
Cabin           object
Embarked        object
dtype: object


As an aside, utilities like cast from visions functional API are simple redirections of an object oriented API defined on the typeset. Feel free to use whichever is most appealing to your personal style.

# Custom Typing

This is all well and good but often we are interested in defining our own custom semantic associations with our data whether as part of an EDA pipeline, a data validation test, or something else altogether and with visions defining those types is easy.

Let’s create a BiologicalSex type corresponding to the same field in our data which has only two unique values: male and female.

from visions.types import VisionsBaseType

class BiologicalSex(VisionsBaseType):
@classmethod
def contains_op(cls, series):
return series.isin({'male', 'female'}).all()

@classmethod
def get_relations(cls):
return []



As we can see, defining a new type is easy, requiring only two methods: contains_op and get_relations. The contains op is a test to determine whether a provided series is of the semantic type represented by the class. In this case, is the data comprised exclusively of the strings male and female.

This simple definition builds a friendly API to work with data. For example, if we wanted to determine whether a sequence was of the type BiologicalSex we can do the following

df.Sex in BiologicalSex

True

df.Age in BiologicalSex

False


What about get_relations though? This method allows us to create relationships between other semantic data types. In this case, our definition of BiologicalSex assumes that all data is a String so let’s now define a relationship between the two and create a new typeset with BiologicalSex.

from visions.relations import IdentityRelation, InferenceRelation
from visions.types import String

class BiologicalSex(VisionsBaseType):
@classmethod
def contains_op(cls, series):
return series.isin({'male', 'female'}).all()

@classmethod
def get_relations(cls):
return [IdentityRelation(cls, String)]

new_typeset = typeset + BiologicalSex
new_typeset.plot_graph()


Voila, like that we have a new type appropriately inserted in the relationship graph of our new typeset and we can do all the same work done above.

# Dealing with Nulls

We do have one potential problem though, we haven’t accounted for any missing value indicators in our contains_op, that means data with missing values won’t be considered to be BiologicalSex. There are numerous ways to resolve this issue (should you wish) but visions offers utility functionality to easily define nullable contains operations.

We can even use both in the same typeset by appropriately defining relations.

null_biosex_series = pd.Series([np.nan, 'male', 'female'])

null_biosex_series in BiologicalSex

False

from visions.utils.series_utils import nullable_series_contains

class NullBiologicalSex(VisionsBaseType):
@classmethod
@nullable_series_contains
def contains_op(cls, series):
return series.isin({'male', 'female'}).all()

@classmethod
def get_relations(cls):
return [IdentityRelation(cls, String)]

class BiologicalSex(VisionsBaseType):
@classmethod
def contains_op(cls, series):
return series.isin({'male', 'female'}).all()

@classmethod
def get_relations(cls):
return [IdentityRelation(cls, NullBiologicalSex, relationship=lambda s: not s.hasnans)]

null_biosex_series in NullBiologicalSex

True


# Bringing it all together

new_typeset = typeset + NullBiologicalSex + BiologicalSex
new_typeset.plot_graph()


import random
new_df = df.astype(str)
new_df['NullSex'] = [random.choice(['male', 'female', np.nan]) for i in range(df.shape[0])]

inferred_types = compare_detect_inference_frame(new_df, new_typeset)
pd.DataFrame({name: [data[i] for data in inferred_types]
for i, name in enumerate(['Columns', 'OriginalType', 'Inferredtype'])})

.dataframe tbody tr th {
vertical-align: top;
}