Python: Summarizing Your Data

Name: Python: Summarizing Your Data
Author: Briana Brownell

Learn techniques for summarizing your data

Preview the first lesson free — get full access to all 9 lessons.

Course: On-Demand

Hi, I'm Brianna Brownell, and in these lessons, we'll teach you how to summarize your data. In this lesson, we're going to review two really important functions on Panda's data frames, the aggregation, and group-by functions. We'll also use them together to get important summary statistics about our data. So again, we have our file loaded, and the first thing that I'm going to do is make sure that the variables that I'm going to use are converted into numerical values rather than categorical. Now, next, what I want to do is a few different summary statistics on the three variables that I've been looking at, the socioeconomic index, the occupational prestige of the respondent's father, and the respondent's occupational prestige. Now, I can use the describe function in order to get a few of the standard values that I might want. But if I want to actually choose the ones that I look at, then I can use this aggregation function on my data frame. So in order to do that, I identify which columns that I want to aggregate, and then I decide which measures I want to use. Here, I've used the mean, the median, and the standard deviation. Now, I can aggregate and print just a single measure, or I can aggregate with as many measures as I want. So let's run this and see what I get. So here, the nice thing about this measure is that I can choose exactly which scores I want to aggregate. So if I'm doing analysis and I have a few really specific things that I want to look at, I can do that with this aggregation. Now, the second thing I want to look at is ways that I can group by a specific variable. In this case, I want to look at a variable that asks respondents how happy they are with their life. And I want to see if there's any relationship between the socio economic index, the occupational prestige, and the occupational prestige of the respondents father in terms of happiness. So the way that I do that is I chain two things. First, I need to select the columns. And so you'll notice that I not only have the numerical columns I want to look at, but I also have the categorical column selected. Now, I need to do that because I'm actually taking a slice of this data frame in order to do this analysis. And so if I leave it out, I'm going to get an error. So then I want to group it by this happiness variable and display the average or the mean. So if I do that, I can see a couple of really interesting things. So the first thing is there does look like there's a relationship between the socio economic index and happiness. Those who are very happy on average have a higher socio economic index compared to those who are not too happy. If I look at the relationship between the averages of the father's occupational prestige, we can see that it doesn't look like there's any real relationship here. Those who are very happy, their fathers have approximately the same occupational prestige as those who are not too happy, for example. And in terms of the individual's occupational prestige, there does look like there is a slight relationship here, but it's really more the socio economic index that's a real indicator. And finally, I can combine these two methods by doing a group by with my categorical variable, and then using the egg method in order to create a number of different measures that I might want to look at. So in this case, you'll see why I might want to do this in a number of different steps, because this table is fairly complicated, and it has a lot of different measures in it. So I may want to look at this table and pick out the things that I think are most important to communicate, and then simply make a table based on those factors. Even so, I can create as large of a table as I want, if I want to look at a number of different summary statistics. In terms of which summary statistics you want to use, this will highly depend on your analysis and the data that you have. There are all kinds of different measures that you may want to use for analysis, and this will depend on your data. But whichever measure you decide to use, the overall procedure for how to summarize your data is still the same. You can use this group by method as well as the aggregation method in order to get the table that you want in order to communicate the results to stakeholders or use to inform the final part of your analysis. In the next lesson, we'll talk a little bit more about how to understand how your data is distributed using a few different visualizations and analysis techniques. Thanks for watching.

Beginner Provider : Briana Brownell 9 Lessons · 53m in English

Course Description

This module will equip you with the knowledge and skills needed to effectively summarize and visualize large and complex data sets. The course covers a wide range of topics, starting with summary statistics, where you will learn how to calculate basic statistical measures such as mean, median, and standard deviation using Python. This will help you to understand the general characteristics of your data.

The module also covers understanding data distributions, including how to visualize data distributions using techniques such as histograms and box plots. Additionally, you will learn about best practices for aggregating data, including choosing appropriate granularity and handling missing values, which will help you to ensure that your data is suitable for analysis. You will also learn about summarizing and visualizing categorical data, using techniques such as counts and proportions. This will help you to understand the distribution of data across different categories, and to identify any patterns or trends. Similarly, you will learn about summarizing and visualizing time-series data using rolling averages, which will help you to identify patterns over time.

The course also covers creating interactive dashboards and visualizations to present data summaries. Additionally, you will learn about summarizing and visualizing relationships between multiple variables, including using techniques such as scatterplots and correlation matrices.

Finally, the course covers common challenges that can arise when summarizing data, such as dealing with missing values, handling outliers, and selecting appropriate summary measures. This will help you to understand how to deal with these common challenges and ensure that your data summaries are accurate and meaningful. Overall, this module will provide you with a comprehensive understanding of how to effectively summarize and visualize data, and it will equip you with the tools and knowledge needed to effectively communicate data insights.

What You'll Learn

Calculate basic statistical measures such as mean, median, and standard deviation using Python
Visualize data distributions using techniques such as histograms and box plots
Apply aggregation best practices, including choosing appropriate granularity and handling missing values
Summarize and visualize categorical data using counts and proportions
Analyze time-series data using rolling averages to identify patterns over time
Build interactive dashboards and explore relationships between variables with scatterplots and correlation matrices

Key Takeaways

Summary statistics like mean, median, and standard deviation help reveal the general characteristics of a data set.
Histograms and box plots are techniques for visualizing how data is distributed.
Choosing appropriate granularity and handling missing values are key best practices when aggregating data.
Categorical data can be summarized with counts and proportions to reveal patterns across categories, while time-series data uses rolling averages to surface patterns over time.
Common summarizing challenges include dealing with missing values, handling outliers, and selecting appropriate summary measures.

Frequently Asked Questions

What will this course teach me?

It equips you with the knowledge and skills to effectively summarize and visualize large and complex data sets, covering summary statistics, data distributions, aggregation best practices, categorical and time-series data, interactive dashboards, multivariate relationships, and common challenges.

What tool is used for the data summarization in this course?

The course teaches you to calculate statistical measures and summarize data using Python.

What topics are covered in the lessons?

Lessons cover Summary Statistics, Understanding Data Distributions, Aggregation Best Practices, Categorical Data, Time-Series Data, Dashboards and Interactivity, Multivariate Data, Data Visualizations, and Common Challenges.

What skills will I gain from this course?

You will build skills in analytics, categorical data, data analysis, data presentation, data visualization, and quantitative data analysis.

How does the course help with presenting data?

It covers creating interactive dashboards and visualizations to present data summaries, and summarizing relationships between multiple variables using techniques such as scatterplots and correlation matrices, helping you effectively communicate data insights.