• From what we've seen so far, it may look like the Series object is basically interchangeable with a one-dimensional NumPy array.

• The essential difference is the presence of the index: while the Numpy Array has an implicitly defined integer index used to access the values, the Pandas Series has an explicitly defined index associated with the values.

• This explicit index definition gives the Series object additional capabilities. For example, the index need not be an integer, but can consist of values of any desired type.

For example, if we wish, we can use strings as an index:

And the item access works as expected:

The dtype object comes from NumPy, it describes the type of element in a ndarray. Every element in a ndarray must has the same size in byte. For int64 and float64, they are 8 bytes. But for strings, the length of the string is not fixed. So instead of save the bytes of strings in the ndarray directly, Pandas use object ndarray, which save pointers to objects, because of this the dtype of this kind ndarray is object.

We can even use non-contiguous or non-sequential indices:

• In this way, you can think of a Pandas Series a bit like a specialization of a Python dictionary.

• A dictionary is a structure that maps arbitrary keys to a set of arbitrary values, and a Series is a structure which maps typed keys to a set of typed values.

• This typing is important: just as the type-specific compiled code behind a NumPy array makes it more efficient than a Python list for certain operations, the type information of a Pandas Series makes it much more efficient than Python dictionaries for certain operations.

The Series-as-dictionary analogy can be made even more clear by constructing a Series object directly from a Python dictionary:

By default, a Series will be created where the index is drawn from the sorted keys.

From here, typical dictionary-style item access can be performed:

Unlike a dictionary, though, the Series also supports array-style operations such as slicing:

We've already seen a few ways of constructing a Pandas Series from scratch; all of them are some version of the following:

where index is an optional argument, and data can be one of many entities.

For example, data can be a list or NumPy array, in which case index defaults to an integer sequence:

data can be a scalar, which is repeated to fill the specified index:

data can be a dictionary, in which index defaults to the dictionary keys:

The index can be explicitly set if a different result is preferred:

Notice that in this case, the Series is populated only with the explicitly identified keys.

Q1: What is the difference between a series and a one dimensional array?

Q2: What is the difference between a series and a dictionary?

• If a Series is an analog of a one-dimensional array with flexible indices, a DataFrame is an analog of a two-dimensional array with both flexible row indices and flexible column names.

• Just as you might think of a two-dimensional array as an ordered sequence of aligned one-dimensional columns, you can think of a DataFrame as a sequence of aligned Series objects. Here, by "aligned" we mean that they share the same index.

To demonstrate this, let's first construct a new Series listing the area of each of the five states discussed in the previous section:

Now that we have this along with the population Series from before, we can use a dictionary to construct a single two-dimensional object containing this information:

Like the Series object, the DataFrame has an index attribute that gives access to the index labels:

Additionally, the DataFrame has a columns attribute, which is an Index object holding the column labels:

Thus the DataFrame can be thought of as a generalization of a two-dimensional NumPy array, where both the rows and columns have a generalized index for accessing the data.

Similarly, we can also think of a DataFrame as a specialization of a dictionary.

• A dictionary maps a key to a value

• A DataFrame maps a column name to a Series of column data.

For example, asking for the 'area' attribute returns the Series object containing the areas we saw earlier:

You can get the specific column as a Series by following code:

A Pandas DataFrame can be constructed in a variety of ways.

Here we'll give several examples.

The Index in many ways operates like an array.

For example, we can use standard Python indexing notation to retrieve values or slices:

Index objects also have many of the attributes familiar from NumPy arrays:

One difference between Index objects and NumPy arrays is that indices are immutable–that is, they cannot be modified via the normal means:

This immutability makes it safer to share indices between multiple DataFrames and arrays, without the potential for side effects from inadvertent index modification.

Pandas objects are designed to facilitate operations such as joins across datasets, which depend on many aspects of set arithmetic.

The Index object follows many of the conventions used by Python's built-in set data structure, so that unions, intersections, differences, and other combinations can be computed in a familiar way:

These operations may also be accessed via object methods, for example indA.intersection(indB).

Like a dictionary, the Series object provides a mapping from a collection of keys to a collection of values:

We can also use dictionary-like Python expressions and methods to examine the keys/indices and values:

Series objects can even be modified with a dictionary-like syntax.

Just as you can extend a dictionary by assigning to a new key, you can extend a Series by assigning to a new index value:

This easy mutability of the objects is a convenient feature: under the hood, Pandas is making decisions about memory layout and data copying that might need to take place; the user generally does not need to worry about these issues.

A Series builds on this dictionary-like interface and provides array-style item selection via the same basic mechanisms as NumPy arrays – that is, slices, masking, and fancy indexing.

Examples of these are as follows:

Notice that when slicing with an explicit index (i.e., data['a':'c']), the final index is included in the slice, while when slicing with an implicit index (i.e., data[0:2]), the final index is excluded from the slice.

These slicing and indexing conventions can be a source of confusion.

For example, if your Series has an explicit integer index, an indexing operation such as data[1] will use the explicit indices, while a slicing operation like data[1:3] will use the implicit Python-style index.

Because of this potential confusion in the case of integer indexes, Pandas provides some special indexer attributes that explicitly expose certain indexing schemes.

These are not functional methods, but attributes that expose a particular slicing interface to the data in the Series.

First, the loc attribute allows indexing and slicing that always references the explicit index:

The iloc attribute allows indexing and slicing that always references the implicit Python-style index:

One guiding principle of Python code is that "explicit is better than implicit."

The explicit nature of loc and iloc make them very useful in maintaining clean and readable code; especially in the case of integer indexes, it is recommended using these both to make code easier to read and understand, and to prevent subtle bugs due to the mixed indexing/slicing convention.

The first analogy we will consider is the DataFrame as a dictionary of related Series objects.

Let's return to our example of areas and populations of states:

The individual Series that make up the columns of the DataFrame can be accessed via dictionary-style indexing of the column name:

Equivalently, we can use attribute-style access with column names that are strings:

This attribute-style column access actually accesses the exact same object as the dictionary-style access:

Though this is a useful shorthand, keep in mind that it does not work for all cases!

For example, if the column names are not strings, or if the column names conflict with methods of the DataFrame, this attribute-style access is not possible.

For example, the DataFrame has a pop() method, so data.pop will point to this rather than the "pop" column:

In particular, you should avoid the temptation to try column assignment via attribute (i.e., use data['pop'] = 1000 rather than data.pop = 1000).

Like with the Series objects discussed earlier, this dictionary-style syntax can also be used to modify the object, in this case adding a new column:

As mentioned previously, we can also view the DataFrame as an enhanced two-dimensional array.

We can examine the raw underlying data array using the values attribute:

With this picture in mind, many familiar array-like observations can be done on the DataFrame itself.

For example, we can transpose the full DataFrame to swap rows and columns:

When it comes to indexing of DataFrame objects, however, it is clear that the dictionary-style indexing of columns precludes our ability to simply treat it as a NumPy array.

In particular, passing a single index to an array accesses a row:

and passing a single "index" to a DataFrame accesses a column:

Thus for array-style indexing, we need another convention.

Here Pandas again uses the loc and iloc indexers mentioned earlier.

Using the iloc indexer, we can index the underlying array as if it is a simple NumPy array (using the implicit Python-style index), but the DataFrame index and column labels are maintained in the result:

Similarly, using the loc indexer we can index the underlying data in an array-like style but using the explicit index and column names:

Any of the familiar NumPy-style data access patterns can be used within these indexers.

For example, in the loc indexer we can combine masking and fancy indexing as in the following:

Any of these indexing conventions may also be used to set or modify values; this is done in the standard way that you might be accustomed to from working with NumPy:

There are a couple extra indexing conventions that might seem at odds with the preceding discussion, but nevertheless can be very useful in practice.

• First, while indexing refers to columns, slicing refers to rows:

Such slices can also refer to rows by number rather than by index:

• Similarly, direct masking operations are also interpreted row-wise rather than column-wise:

These two conventions are syntactically similar to those on a NumPy array, and while these may not precisely fit the mold of the Pandas conventions, they are nevertheless quite useful in practice.

As an example, suppose we are combining two different data sources, and find only the top three US states by area and the top three US states by population:

Let's see what happens when we divide these to compute the population density:

The resulting array contains the union of indices of the two input arrays.

Any item for which one or the other does not have an entry is marked with NaN, or "Not a Number," which is how Pandas marks missing data.

This index matching is implemented this way for any of Python's built-in arithmetic expressions; any missing values are filled in with NaN by default:

If using NaN values is not the desired behavior, the fill value can be modified using appropriate object methods in place of the operators.

For example, calling A.add(B) is equivalent to calling A + B, but allows optional explicit specification of the fill value for any elements in A or B that might be missing:

A similar type of alignment takes place for both columns and indices when performing operations on DataFrames:

Notice that indices are aligned correctly irrespective of their order in the two objects, and indices in the result are sorted.

As was the case with Series, we can use the associated object's arithmetic method and pass any desired fill_value to be used in place of missing entries.

Here we'll fill with the mean of all values in A (computed by first stacking the rows of A):

The following table lists Python operators and their equivalent Pandas object methods:

| Python Operator | Pandas Method(s) |

|-----------------|---------------------------------------|

| + | add() |

| - | sub(), subtract() |

| * | mul(), multiply() |

| / | truediv(), div(), divide()|

| // | floordiv() |

| % | mod() |

| ** | pow() |

• df.info()

• df.shape (similar to "dim" in R)

• df.columns (check the variables, similar to "names" in R)

• df.index (check the index of the "rows")

• df.describe() (descriptive statistics for numerical variables)

• Note that methods (functions) have parentheses, and attributes (values) don't.