- doctorsmonsters

# Basic Data Exploration in Pandas Dataframe

In this exercise today, we will perform some simple data exploration using pandas in python. We will use a dataset that has information about various car models. The data is in a CSV file, mtcars.csv.

The notebook for this tutorial along with the dataset can be found __here__.

We can start by importing pandas and loading the data into the dataframe.

```
import pandas as pd
data = pd.read_csv('mtcars.csv')
```

Now that we have our data in a dataframe, we can take a peak into the data.

`data.head()`

We can also quickly get some statistics on the data by using the describe function.

`data.describe()`

We can also get information about the columns and datatypes of each column and the count of non-null values.

`data.info()`

```
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 32 entries, 0 to 31
Data columns (total 12 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 model 32 non-null object
1 mpg 32 non-null float64
2 cyl 32 non-null int64
3 disp 32 non-null float64
4 hp 32 non-null int64
5 drat 32 non-null float64
6 wt 32 non-null float64
7 qsec 32 non-null float64
8 vs 32 non-null int64
9 am 32 non-null int64
10 gear 32 non-null int64
11 carb 32 non-null int64
dtypes: float64(5), int64(6), object(1)
memory usage: 3.1+ KB
```

We can also look for null values in the dataframe.

`data.isnull().sum()`

```
model 0
mpg 0
cyl 0
disp 0
hp 0
drat 0
wt 0
qsec 0
vs 0
am 0
gear 0
carb 0
dtype: int64
```

Now lets say we want to see which model has the maximum MPG. We can do that by finding the row in which the mpg column has the highest values.

`data.loc[data['mpg'].idxmax()]`

```
model Toyota Corolla
mpg 33.9
cyl 4
disp 71.1
hp 65
drat 4.22
wt 1.835
qsec 19.9
vs 1
am 1
gear 4
carb 1
Name: 19, dtype: object
```

As we can see, Toyota Corolla has the highest MPG in our dataset. If we are only interested in the name of the model, we can modify the code above by adding the name of the desired column.

`data.loc[data['mpg'].idxmax()]['model']`

`'Toyota Corolla'`

Similarly, if we want value from more than one columns to be displayed, we can do that by passing the names of the columns as a list.

`data.loc[data['mpg'].idxmax()][['model','wt','qsec']]`

```
model Toyota Corolla
wt 1.835
qsec 19.9
Name: 19, dtype: object
```

The opposite of idxmin, so to get a minimum value in a columns, you can use the above code but replace idxmax with idxmin.

Another step in data exploration is correlation between variables. Pandas makes that very easy. We can have it draw a correlation matrix to give us a broad sense of correlation between different variables.

`data.corr()`

If you are interested in correlation between only two variables, for example, between mpg and wt, we can calculate that as following.

`data.mpg.corr(data.wt)`

`-0.8676593765172279`

Now what if we want to look at correlations of only one variable with all of the other variables. This is simple. First we draw the correlation matrix as mentioned above, cast it on variable to store it, and then can retrieve correlation of any of the columns with others.

`matrix = data.corr()`

`matrix['mpg']`

```
mpg 1.000000
cyl -0.852162
disp -0.847551
hp -0.776168
drat 0.681172
wt -0.867659
qsec 0.418684
vs 0.664039
am 0.599832
gear 0.480285
carb -0.550925
Name: mpg, dtype: float64
```

We can also sort these correlation values so that we can see which variables have the most effect on mpg in descending order.

`matrix['mpg'].sort_values(ascending = False)`

```
mpg 1.000000
drat 0.681172
vs 0.664039
am 0.599832
gear 0.480285
qsec 0.418684
carb -0.550925
hp -0.776168
disp -0.847551
cyl -0.852162
wt -0.867659
Name: mpg, dtype: float64
```

By default, the sort_values function with display values in an ascending order therefore we set it to False to get values in a descending order.

It would not be fair to talk about correlation without talking about covariance.

Covariance determines how much a variable changes with a change in the other variable. It could be positive of negative. Positive covariance means that the variables will change in the same direction. If it is negative, the variables move in opposite directions.

The syntax for covariance is similar to correlation, but replace corr with cov. So to get a covariance matrix, you can simply use:

`data.cov()`

All the other tasks that we did with correlation can be done for covariance as well.