Python: Discovering Common Relationships in Data

Name: Python: Discovering Common Relationships in Data
Author: Briana Brownell

Learn how to discover common relationships in data

Preview the first lesson free — get full access to all 6 lessons.

Course: On-Demand

Hi, I'm Brianna Brownell, and in these lessons we're going to talk about discovering common relationships in data, in this lesson where we will first look at exploratory analysis and how you can find out basic information about a data set using the Pandas library. So here again we have our Jupyter Notebook and I'm going to review the code in these cells and we can follow along. So the first cell has all of my setup which includes installing the libraries that I need and allowing me to upload the file that I want to work with. So I can upload the file that I want. You'll see here that I am importing NumPy as NP and so this library we will come across a little bit later in this lesson because we'll need it to do a few different things. So the next thing that I'm going to do is I'm going to read that data file into the data frame and do some conversions of the different variables. So remember how we had a whole bunch of our columns ended up being categorical. Well we would like to correct that and treat them as floating point or integer numbers. So I'm going to run that and then I'm going to use the describe function to give us some information about those variables. So here I've got three different variables that are kind of interesting to talk about. So the first one is the socio-economic index. The second one is the occupational prestige score. The third one is the occupational prestige score of the respondents father. So the describe function gives us a number of different values including the number of valid cases, the average, the minimum value, and then a few different percentiles. The 25th percentile, the median or 50th percentile, the 75th percentile, and then the maximum value. So in order to understand what these scores are we need to go back to our documentation on the general social survey. Okay so this gives us a little bit of information about the occupational prestige score and the socio-economic index scores that are given in the survey. So for the SCI score essentially it's a combination of both the occupation title as well as education and income. Meanwhile the occupational prestige score is based on data that's collected about what people's perceptions are of the social standing of various occupations. As you can see there are a huge number of occupations listed. So by reading the documentation we can get a better idea of what these variables mean and how we can use them. Okay so let's go back to looking at this variable adults. So the adults variable tells us how many members of the household who are 18 years old or older. So let's see how that data is distributed. So you can see right away that we have an average of just under two adults per household, the minimum being one and the maximum being seven. The median in this case is two adults per household. Now this all completely makes sense and we can sort of infer the distribution by looking at these variables and noticing that this 25th percentile is one adult per household and the median is two per household. So there are a majority of households that have one or two adults per household and this completely makes sense. The other thing that we might want to do is we might want to sort the values of the data set and we can do that by using the sort values function. So if I do that then I can look at the head of the data set again. Here I've sorted by occupational prestige and it's interesting because we can look at this occupation code and see that the individuals in this data set who have the highest occupational prestige are physicians and surgeons. It's important to do some sorting and analysis like that in order to do a gut check of the results. And so if I want to try to make sure that I understand occupational prestige then I can sort by that score and see whether the answers make sense. And in this case I think that it does. So another thing that we can do is do some basic visualizations of the data to see what the distribution looks like. Here we have a histogram of the socioeconomic index score and basically I've used a combination of numpy and matplotlib in order to visualize a histogram and so we can see what the distribution looks like. So thinking about the actual distribution of this it looks like it's sort of a skewed distribution so it's not a normal distribution like we might expect. So there are a lot of individuals who are sort of in this 22 20 to 40 sort of range. So this will be important when we're doing our analysis and trying to understand what would be a typical score. Another thing that we can do with specific columns in our data set is we can do mathematical operations on them. So in this case maybe we want to know the total number of adults that are represented in all of the households in our data set. So here we can sum that column and get our result. So here we see we have just over 3000 adults in our data set. So this is another thing where it's good to do some of this analysis in order to do a gut check of the results. So remember we had about 1600 people in our survey. So the size of our data set was about 1600 rows long and we saw from doing our analysis on the adults column that our average was just under two. So we would expect it to be not quite double the count as the total number of adults that are represented in all of the households in our data set. So making sure that you stop and think about what the values are that are coming out of the data is a really important part of exploratory analysis. And this is for a couple reasons. The first one is to make sure that you as the analyst understand the data that you are analyzing. So you understand approximately the ranges of the scores, you understand what the variables are measuring, and to make sure that the way that you're interpreting the results are accurate to the data. So it's a good best practice to always do some exploratory analysis on a data set when you get it in order to get a sense of the data and a feel for the results that it contains. Now this data set is relatively small. We have really no trouble doing the analysis with it. So this is a good thing, but often you have a very large data set and instead of doing the analysis on the entire data set, in order to effectively explore it, you want to just take a sample of that data set. So we can also do that in Pandas. So here we're sampling the data frame with 100 rows. And so if I do that, then I can see that now when it sort of prints the summary of this data set, it's 100 rows by that 703 columns instead of our 1600 as we had before. We can also sample by a fraction of the data set. So this will be important when we want to build models on data and instead of building it on the entire data set, we want to build it on a training set and then test the quality of the model on a test set. In the next lesson, we'll talk about business intelligence. Thanks for watching.

Beginner Provider : Briana Brownell 6 Lessons · 40m in English

Course Description

This module will teach you how to explore data, and how to understand and communicate analysis results using visualizations and statistics. The course begins with exploratory analysis, where you will learn how to use visualizations and statistics to explore and understand patterns in data. This includes techniques such as histograms, scatter plots, and box plots, which will help you to identify outliers, trends, and patterns in the data. You will also learn about business intelligence and how to use data analysis to support business decision-making. This includes the use of dashboards and visualizations to communicate insights to stakeholders and how to use data to identify opportunities for growth and improvement.

The course also covers important preprocessing steps necessary before conducting statistical analysis. This includes cleaning and preprocessing data, handling missing values, and ensuring that the data is suitable for analysis. You will learn about various techniques for transforming data, such as scaling, normalization, and aggregation, which will help you to prepare your data for analysis.

Additionally, the course covers Correlations and Causality, where you will learn how to identify and interpret correlations in data, including the distinction between correlations and causality. Finally, you will learn techniques to compare groups of data. This includes the use of crosstabulations and statistical tests. You’ll be equipped with the tools and knowledge needed to effectively explore, understand, and communicate patterns in data, and to support business decision-making using data analysis.

What You'll Learn

Apply exploratory data analysis techniques such as histograms, scatter plots, and box plots to identify outliers, trends, and patterns
Analyze data to support business decision-making using dashboards and visualizations that communicate insights to stakeholders
Understand preprocessing steps including cleaning data, handling missing values, and ensuring data is suitable for analysis
Transform data using techniques such as scaling, normalization, and aggregation to prepare it for analysis
Identify and interpret correlations in data, including the distinction between correlation and causality
Compare groups of data using crosstabulations and statistical tests

Key Takeaways

Exploratory analysis uses visualizations and statistics to explore and understand patterns in data.
Business intelligence applies data analysis to support decision-making and to identify opportunities for growth and improvement.
Preprocessing steps such as cleaning data and handling missing values ensure that data is suitable for analysis.
Correlations can be identified and interpreted, but correlation is distinct from causality.
Groups of data can be compared using crosstabulations and statistical tests.

Frequently Asked Questions

What does this course cover?

The course covers exploratory analysis, business intelligence, preprocessing steps, transforming data, correlations and causality, and comparing groups of data using crosstabulations and statistical tests.

What will I learn to do with data exploration?

You will learn to use visualizations and statistics, including histograms, scatter plots, and box plots, to identify outliers, trends, and patterns in data.

How does this course relate to business decisions?

It teaches how to use data analysis to support business decision-making, including using dashboards and visualizations to communicate insights to stakeholders and identify opportunities for growth and improvement.

What data preparation techniques are taught?

The course covers cleaning and preprocessing data, handling missing values, and transforming data through scaling, normalization, and aggregation to prepare it for analysis.

What skills does this course build?

It builds skills in analytics, business analytics, data analysis, data mining, data visualization, and quantitative data analysis.