[2023 update] Introduction to Polars
It has been over 2 years already since I created my first notebook about Polars (Medium post here). Back then a small project started by Ritchie Vink with promising performance. 2 years later, the project has over 10.000 stars on Github and is still leading many performance benchmarks.
In this updated tutorial we will revisit the original analyses and rewrite the queries following the latest version of Polars and the Polars book.
Find the accompanying notebook here to tweak and run all the examples.
Small reading guide
Text blocks in quotes are coming from the first notebook.
Installing Polars
The first notebook ran with Polars 0.7.0. Definitely time for an update.
First install Polars in your virtual environment. In this notebook we will work with Polars version 0.15.7. Add this to the pip install to ensure all cells run.
pip install polars==0.15.7Importing relevant packages and loading the data
Polars already offers many functionalities that we are already familiar if you have worked with Pandas before. We can find an overview, including examples (for most), in the reference guide.
Find the dataset that we are going to use here: Winemag-dataset
Basic Dataframe inspection
Nothing has changed so far, until now. Instead of the Pandas-like syntax we used two years ago. We will use the expressions, as recommended by Polars.
We strongly recommend selecting data with expressions for almost all use cases. Square bracket indexing is perhaps useful when doing exploratory data analysis in a terminal or notebook when you just want a quick look at a subset of data. ~ Polars User Guide
shape: (5, 11)
┌────────┬───────────┬───────────┬───────────┬─────┬───────────┬───────────┬───────────┬───────────┐
│ ┆ country ┆ descripti ┆ designati ┆ ... ┆ region_1 ┆ region_2 ┆ variety ┆ winery │
│ --- ┆ --- ┆ on ┆ on ┆ ┆ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ str ┆ --- ┆ --- ┆ ┆ str ┆ str ┆ str ┆ str │
│ ┆ ┆ str ┆ str ┆ ┆ ┆ ┆ ┆ │
╞════════╪═══════════╪═══════════╪═══════════╪═════╪═══════════╪═══════════╪═══════════╪═══════════╡
│ 91868 ┆ US ┆ Beautiful ┆ Elaine's ┆ ... ┆ Finger ┆ Finger ┆ Pinot ┆ Heart & │
│ ┆ ┆ ly ┆ Vineyard ┆ ┆ Lakes ┆ Lakes ┆ Noir ┆ Hands │
│ ┆ ┆ fragrant ┆ ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ of sweet ┆ ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ bl... ┆ ┆ ┆ ┆ ┆ ┆ │
├╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┤
│ 102345 ┆ Argentina ┆ This is ┆ Reservado ┆ ... ┆ San Juan ┆ null ┆ Torrontés ┆ Callia │
│ ┆ ┆ neutral ┆ ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ on the ┆ ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ nose ┆ ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ exce... ┆ ┆ ┆ ┆ ┆ ┆ │
├╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┤
│ 16133 ┆ Austria ┆ A clean, ┆ Classic ┆ ... ┆ null ┆ null ┆ Welschrie ┆ Adolf & │
│ ┆ ┆ fruity ┆ ┆ ┆ ┆ ┆ sling ┆ Heinrich │
│ ┆ ┆ wine, ┆ ┆ ┆ ┆ ┆ ┆ Fuchs │
│ ┆ ┆ with ┆ ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ flavo... ┆ ┆ ┆ ┆ ┆ ┆ │
├╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┤
...
│ ┆ ┆ Villa ┆ Novare ┆ ┆ lla ┆ ┆ Rondinell ┆ │
│ ┆ ┆ Novare is ┆ ┆ ┆ Superiore ┆ ┆ a, ┆ │
│ ┆ ┆ a nice... ┆ ┆ ┆ Ripasso ┆ ┆ Molinara ┆ │
└────────┴───────────┴───────────┴───────────┴─────┴───────────┴───────────┴───────────┴───────────┘
@r-brinkWe could still use the old way, as it is shorter. However, we can easily extend this expression to include more information if we want. It also helps us to solidify to work with expression in our workflow.
The dataset has a lot to offer. With 11 variables and over 150k rows, there is a lot of data to analyse. We see a couple of variables that are interesting to look into, like
price,country,points.
To select a column, we use the following expression.
Removing nulls
Before we continue we want to have a closer look at if there are any
nullsin the dataset.
Similar to our original notebook we will fill the nulls with the mean for the following reason.
It seems that around a little less than 10% of the price variable has no value. We can either drop the rows with missing values or fill them. In this article, we will choose to use the mean as filling strategy.
Some analyses
The next step is to dive in a little deeper and have a closer look at the dataset with some more complex functions.
The goal that we want to achieve in the following part is to have a closer look at the countries and how they compare in terms of price and points.
Or we can just use the describe() function for quick statistics about our Dataframe.
shape: (7, 12)
┌───────────┬───────────┬─────────┬───────────┬─────┬───────────┬───────────┬───────────┬──────────┐
│ describe ┆ column_0 ┆ country ┆ descripti ┆ ... ┆ region_1 ┆ region_2 ┆ variety ┆ winery │
│ --- ┆ --- ┆ --- ┆ on ┆ ┆ --- ┆ --- ┆ --- ┆ --- │
│ str ┆ f64 ┆ str ┆ --- ┆ ┆ str ┆ str ┆ str ┆ str │
│ ┆ ┆ ┆ str ┆ ┆ ┆ ┆ ┆ │
╞═══════════╪═══════════╪═════════╪═══════════╪═════╪═══════════╪═══════════╪═══════════╪══════════╡
│ count ┆ 150924.0 ┆ 150924 ┆ 150924 ┆ ... ┆ 150924 ┆ 150924 ┆ 150924 ┆ 150924 │
├╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┤
│ null_coun ┆ 0.0 ┆ 0 ┆ 0 ┆ ... ┆ 25054 ┆ 89971 ┆ 0 ┆ 0 │
│ t ┆ ┆ ┆ ┆ ┆ ┆ ┆ ┆ │
├╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┤
│ mean ┆ 75464.428 ┆ null ┆ null ┆ ... ┆ null ┆ null ┆ null ┆ null │
│ ┆ 587 ┆ ┆ ┆ ┆ ┆ ┆ ┆ │
├╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┤
│ std ┆ 43569.166 ┆ null ┆ null ┆ ... ┆ null ┆ null ┆ null ┆ null │
│ ┆ 909 ┆ ┆ ┆ ┆ ┆ ┆ ┆ │
├╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┤
│ min ┆ 0.0 ┆ Albania ┆ "Chremisa ┆ ... ┆ Abruzzo ┆ Californi ┆ Agiorgiti ┆ '37 │
│ ┆ ┆ ┆ ," the ┆ ┆ ┆ a Other ┆ ko ┆ Cellars │
│ ┆ ┆ ┆ ancient ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ ┆ name of ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ ┆ ... ┆ ┆ ┆ ┆ ┆ │
├╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┤
│ max ┆ 150929.0 ┆ Uruguay ┆ “Zéro ┆ ... ┆ Zonda ┆ Willamett ┆ Žilavka ┆ Štoka │
│ ┆ ┆ ┆ Dosage” ┆ ┆ Valley ┆ e Valley ┆ ┆ │
│ ┆ ┆ ┆ means ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ ┆ just ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ ┆ that; ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ ┆ t... ┆ ┆ ┆ ┆ ┆ │
├╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┤
│ median ┆ 75464.5 ┆ null ┆ null ┆ ... ┆ null ┆ null ┆ null ┆ null │
└───────────┴───────────┴─────────┴───────────┴─────┴───────────┴───────────┴───────────┴──────────┘
@r-brinkThe minimum number of points shows that there is no such thing as bad wine.
Filtering
From our earlier notebook
There are two strange values in our dataset: an undefined country (“”) and a country called ‘US-France’.
shape: (6, 11)
┌────────┬───────────┬───────────┬───────────┬─────┬──────────┬──────────┬─────────────┬───────────┐
│ ┆ country ┆ descripti ┆ designati ┆ ... ┆ region_1 ┆ region_2 ┆ variety ┆ winery │
│ --- ┆ --- ┆ on ┆ on ┆ ┆ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ str ┆ --- ┆ --- ┆ ┆ str ┆ str ┆ str ┆ str │
│ ┆ ┆ str ┆ str ┆ ┆ ┆ ┆ ┆ │
╞════════╪═══════════╪═══════════╪═══════════╪═════╪══════════╪══════════╪═════════════╪═══════════╡
│ 1133 ┆ null ┆ Delicate ┆ Askitikos ┆ ... ┆ null ┆ null ┆ Assyrtiko ┆ Tsililis │
│ ┆ ┆ white ┆ ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ flowers ┆ ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ and a ┆ ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ spi... ┆ ┆ ┆ ┆ ┆ ┆ │
├╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┤
│ 1440 ┆ null ┆ A blend ┆ Shah ┆ ... ┆ null ┆ null ┆ Red Blend ┆ Büyülübağ │
│ ┆ ┆ of 60% ┆ ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ Syrah, ┆ ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ 30% ┆ ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ Cabern... ┆ ┆ ┆ ┆ ┆ ┆ │
├╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┤
│ 68226 ┆ null ┆ From ┆ Piedra ┆ ... ┆ null ┆ null ┆ Pinot Noir ┆ Chilcas │
│ ┆ ┆ first ┆ Feliz ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ sniff to ┆ ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ last, the ┆ ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ no... ┆ ┆ ┆ ┆ ┆ ┆ │
├╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┤
│ 113016 ┆ null ┆ From ┆ Piedra ┆ ... ┆ null ┆ null ┆ Pinot Noir ┆ Chilcas │
│ ┆ ┆ first ┆ Feliz ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ sniff to ┆ ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ last, the ┆ ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ no... ┆ ┆ ┆ ┆ ┆ ┆ │
├╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┤
│ 135696 ┆ null ┆ From ┆ Piedra ┆ ... ┆ null ┆ null ┆ Pinot Noir ┆ Chilcas │
│ ┆ ┆ first ┆ Feliz ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ sniff to ┆ ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ last, the ┆ ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ no... ┆ ┆ ┆ ┆ ┆ ┆ │
├╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┤
│ 144054 ┆ US-France ┆ Defies ┆ null ┆ ... ┆ null ┆ null ┆ Viognier ┆ Deux C │
│ ┆ ┆ categoriz ┆ ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ ation, in ┆ ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ more w... ┆ ┆ ┆ ┆ ┆ ┆ │
└────────┴───────────┴───────────┴───────────┴─────┴──────────┴──────────┴─────────────┴───────────┘
@r-brinkThere are only 6 of them, so it is safe to drop them.
Time to look into the countries that produces the best wine according to the points and has the highest price for a bottle.
England is leading the list for the best wines. Wonder how they think about that on the other side of the Canal in France.
Plotting with Polars
To get a better insight into the differences it always helps to have some nice plots. Where Pandas has a plotting functionality build in, we have to rely on our Matplotlib skills for Polars. We focus on the top 15 countries.
shape: (123148, 12)
┌─────────┬───────────┬───────┬───────────┬─────┬──────────┬──────────┬──────────────┬─────────────┐
│ country ┆ points_me ┆ ┆ descripti ┆ ... ┆ region_1 ┆ region_2 ┆ variety ┆ winery │
│ --- ┆ an ┆ --- ┆ on ┆ ┆ --- ┆ --- ┆ --- ┆ --- │
│ str ┆ --- ┆ i64 ┆ --- ┆ ┆ str ┆ str ┆ str ┆ str │
│ ┆ f64 ┆ ┆ str ┆ ┆ ┆ ┆ ┆ │
╞═════════╪═══════════╪═══════╪═══════════╪═════╪══════════╪══════════╪══════════════╪═════════════╡
│ England ┆ 92.888889 ┆ 28800 ┆ Gentle ┆ ... ┆ null ┆ null ┆ Chardonnay ┆ Hattingley │
│ ┆ ┆ ┆ white ┆ ┆ ┆ ┆ ┆ Valley │
│ ┆ ┆ ┆ blossom, ┆ ┆ ┆ ┆ ┆ Wines │
│ ┆ ┆ ┆ white ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ ┆ curr... ┆ ┆ ┆ ┆ ┆ │
├╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ England ┆ 92.888889 ┆ 28801 ┆ A lifting ┆ ... ┆ null ┆ null ┆ Sparkling ┆ Ridgeview │
│ ┆ ┆ ┆ mix of ┆ ┆ ┆ ┆ Blend ┆ Estate │
│ ┆ ┆ ┆ golden, ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ ┆ ripe ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ ┆ ap... ┆ ┆ ┆ ┆ ┆ │
├╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ England ┆ 92.888889 ┆ 28818 ┆ Sophistic ┆ ... ┆ null ┆ null ┆ Chardonnay ┆ Wiston │
│ ┆ ┆ ┆ ated ┆ ┆ ┆ ┆ ┆ Estate │
│ ┆ ┆ ┆ aromas of ┆ ┆ ┆ ┆ ┆ Winery │
│ ┆ ┆ ┆ Golden ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ ┆ P... ┆ ┆ ┆ ┆ ┆ │
├╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ England ┆ 92.888889 ┆ 28822 ┆ Both the ┆ ... ┆ null ┆ null ┆ Sparkling ┆ Wiston │
│ ┆ ┆ ┆ fruit and ┆ ┆ ┆ ┆ Blend ┆ Estate │
│ ┆ ┆ ┆ the ┆ ┆ ┆ ┆ ┆ Winery │
│ ┆ ┆ ┆ aromatic ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ ┆ ... ┆ ┆ ┆ ┆ ┆ │
├╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ ... ┆ ... ┆ ... ┆ ... ┆ ... ┆ ... ┆ ... ┆ ... ┆ ... │
├╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ India ┆ 87.625 ┆ 3840 ┆ Pineapple ┆ ... ┆ null ┆ null ┆ Chenin Blanc ┆ Sula │
│ ┆ ┆ ┆ , grapefr ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ ┆ uit, and ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ ┆ apric... ┆ ┆ ┆ ┆ ┆ │
├╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ India ┆ 87.625 ┆ 7804 ┆ This wine ┆ ... ┆ null ┆ null ┆ Chenin Blanc ┆ Sula │
│ ┆ ┆ ┆ features ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ ┆ a fresh ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ ┆ nose ... ┆ ┆ ┆ ┆ ┆ │
├╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ India ┆ 87.625 ┆ 36579 ┆ Charred ┆ ... ┆ null ┆ null ┆ Cabernet ┆ Good Earth │
│ ┆ ┆ ┆ wood and ┆ ┆ ┆ ┆ Sauvignon ┆ Wine │
│ ┆ ┆ ┆ smoke ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ ┆ dominate ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ ┆ ... ┆ ┆ ┆ ┆ ┆ │
├╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ India ┆ 87.625 ┆ 80529 ┆ Charred ┆ ... ┆ null ┆ null ┆ Cabernet ┆ Good Earth │
│ ┆ ┆ ┆ wood and ┆ ┆ ┆ ┆ Sauvignon ┆ Wine │
│ ┆ ┆ ┆ smoke ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ ┆ dominate ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ ┆ ... ┆ ┆ ┆ ┆ ┆ │
└─────────┴───────────┴───────┴───────────┴─────┴──────────┴──────────┴──────────────┴─────────────┘
@r-brink
Time to go Lazy
The lazy API offers a way to optimise your queries, similar to Spark. The major benefit over spark is that we don’t have to set up our environment and can therefore continue working from our notebook.
More information can be found in the [Polars-book](https://ritchie46.github.io/polars-book/lazy_polars/intro.html)
Printing the type returns ‘polars.lazy.LazyFrame’ indicating the data is available to us. On to the Groupby `country` and find the average `points` to compare with the eager API that we used earlier.
Similar to the filters that we did with the eager API we are going to filter the unknown and ‘US-France’ values in the `country` variable first.
We can see that the query is almost the same, however this query only returns a query plan.
As we can see nothing happens right away. From the documentation: ‘This is due to the lazyness, nothing will happen until specifically requested. This allows Polars to see the whole context of a query and optimize just in time for execution.’
To actually see the results we can do two things:
collect()andfetch(). The difference is thatfetchtakes the first 500 rows and then runs the query, whereascollectruns the query over all the results. Below we can see the differences for our case.
shape: (500, 11)
┌─────┬─────────┬───────────┬───────────┬─────┬─────────────┬────────────┬────────────┬────────────┐
│ ┆ country ┆ descripti ┆ designati ┆ ... ┆ region_1 ┆ region_2 ┆ variety ┆ winery │
│ --- ┆ --- ┆ on ┆ on ┆ ┆ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ str ┆ --- ┆ --- ┆ ┆ str ┆ str ┆ str ┆ str │
│ ┆ ┆ str ┆ str ┆ ┆ ┆ ┆ ┆ │
╞═════╪═════════╪═══════════╪═══════════╪═════╪═════════════╪════════════╪════════════╪════════════╡
│ 0 ┆ US ┆ This trem ┆ Martha's ┆ ... ┆ Napa Valley ┆ Napa ┆ Cabernet ┆ Heitz │
│ ┆ ┆ endous ┆ Vineyard ┆ ┆ ┆ ┆ Sauvignon ┆ │
│ ┆ ┆ 100% ┆ ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ varietal ┆ ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ wi... ┆ ┆ ┆ ┆ ┆ ┆ │
├╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┤
│ 1 ┆ Spain ┆ Ripe ┆ Carodorum ┆ ... ┆ Toro ┆ null ┆ Tinta de ┆ Bodega │
│ ┆ ┆ aromas of ┆ Selección ┆ ┆ ┆ ┆ Toro ┆ Carmen │
│ ┆ ┆ fig, blac ┆ Especial ┆ ┆ ┆ ┆ ┆ Rodríguez │
│ ┆ ┆ kberry ┆ Res... ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ a... ┆ ┆ ┆ ┆ ┆ ┆ │
├╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┤
│ 2 ┆ US ┆ Mac ┆ Special ┆ ... ┆ Knights ┆ Sonoma ┆ Sauvignon ┆ Macauley │
│ ┆ ┆ Watson ┆ Selected ┆ ┆ Valley ┆ ┆ Blanc ┆ │
│ ┆ ┆ honors ┆ Late ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ the ┆ Harvest ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ memory of ┆ ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ ... ┆ ┆ ┆ ┆ ┆ ┆ │
├╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┤
│ 3 ┆ US ┆ This ┆ Reserve ┆ ... ┆ Willamette ┆ Willamette ┆ Pinot Noir ┆ Ponzi │
│ ┆ ┆ spent 20 ┆ ┆ ┆ Valley ┆ Valley ┆ ┆ │
│ ┆ ┆ months in ┆ ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ 30% new ┆ ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ ... ┆ ┆ ┆ ┆ ┆ ┆ │
├╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┤
│ ... ┆ ... ┆ ... ┆ ... ┆ ... ┆ ... ┆ ... ┆ ... ┆ ... │
├╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┤
│ 496 ┆ US ┆ Boysenber ┆ Thompson ┆ ... ┆ Santa ┆ Central ┆ Syrah ┆ Rideau │
│ ┆ ┆ ry, dried ┆ Vineyard ┆ ┆ Barbara ┆ Coast ┆ ┆ │
│ ┆ ┆ violets ┆ ┆ ┆ County ┆ ┆ ┆ │
│ ┆ ┆ and b... ┆ ┆ ┆ ┆ ┆ ┆ │
├╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┤
│ 497 ┆ Italy ┆ This ┆ Daniello ┆ ... ┆ Toscana ┆ null ┆ Red Blend ┆ Tenuta di │
│ ┆ ┆ blend of ┆ ┆ ┆ ┆ ┆ ┆ Trecciano │
│ ┆ ┆ 80% ┆ ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ Cabernet ┆ ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ Sauvi... ┆ ┆ ┆ ┆ ┆ ┆ │
├╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┤
│ 498 ┆ Italy ┆ This ┆ Il ┆ ... ┆ Toscana ┆ null ┆ Cabernet ┆ Terre del │
│ ┆ ┆ brooding ┆ Tarabuso ┆ ┆ ┆ ┆ Sauvignon ┆ Marchesato │
│ ┆ ┆ wine ┆ ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ opens ┆ ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ with ┆ ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ ar... ┆ ┆ ┆ ┆ ┆ ┆ │
├╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┤
│ 499 ┆ Spain ┆ Earthy ┆ null ┆ ... ┆ Priorat ┆ null ┆ Red Blend ┆ Vega Escal │
│ ┆ ┆ berry and ┆ ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ licorice ┆ ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ aromas... ┆ ┆ ┆ ┆ ┆ ┆ │
└─────┴─────────┴───────────┴───────────┴─────┴─────────────┴────────────┴────────────┴────────────┘
@r-brinkshape: (150924, 11)
┌────────┬─────────┬───────────┬───────────┬─────┬────────────┬────────────┬───────────┬───────────┐
│ ┆ country ┆ descripti ┆ designati ┆ ... ┆ region_1 ┆ region_2 ┆ variety ┆ winery │
│ --- ┆ --- ┆ on ┆ on ┆ ┆ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ str ┆ --- ┆ --- ┆ ┆ str ┆ str ┆ str ┆ str │
│ ┆ ┆ str ┆ str ┆ ┆ ┆ ┆ ┆ │
╞════════╪═════════╪═══════════╪═══════════╪═════╪════════════╪════════════╪═══════════╪═══════════╡
│ 0 ┆ US ┆ This trem ┆ Martha's ┆ ... ┆ Napa ┆ Napa ┆ Cabernet ┆ Heitz │
│ ┆ ┆ endous ┆ Vineyard ┆ ┆ Valley ┆ ┆ Sauvignon ┆ │
│ ┆ ┆ 100% ┆ ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ varietal ┆ ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ wi... ┆ ┆ ┆ ┆ ┆ ┆ │
├╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┤
│ 1 ┆ Spain ┆ Ripe ┆ Carodorum ┆ ... ┆ Toro ┆ null ┆ Tinta de ┆ Bodega │
│ ┆ ┆ aromas of ┆ Selección ┆ ┆ ┆ ┆ Toro ┆ Carmen │
│ ┆ ┆ fig, blac ┆ Especial ┆ ┆ ┆ ┆ ┆ Rodríguez │
│ ┆ ┆ kberry ┆ Res... ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ a... ┆ ┆ ┆ ┆ ┆ ┆ │
├╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┤
│ 2 ┆ US ┆ Mac ┆ Special ┆ ... ┆ Knights ┆ Sonoma ┆ Sauvignon ┆ Macauley │
│ ┆ ┆ Watson ┆ Selected ┆ ┆ Valley ┆ ┆ Blanc ┆ │
│ ┆ ┆ honors ┆ Late ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ the ┆ Harvest ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ memory of ┆ ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ ... ┆ ┆ ┆ ┆ ┆ ┆ │
├╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┤
│ 3 ┆ US ┆ This ┆ Reserve ┆ ... ┆ Willamette ┆ Willamette ┆ Pinot ┆ Ponzi │
│ ┆ ┆ spent 20 ┆ ┆ ┆ Valley ┆ Valley ┆ Noir ┆ │
│ ┆ ┆ months in ┆ ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ 30% new ┆ ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ ... ┆ ┆ ┆ ┆ ┆ ┆ │
├╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┤
│ ... ┆ ... ┆ ... ┆ ... ┆ ... ┆ ... ┆ ... ┆ ... ┆ ... │
├╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┤
│ 150926 ┆ France ┆ Offers an ┆ Cuvée ┆ ... ┆ Champagne ┆ null ┆ Champagne ┆ H.Germain │
│ ┆ ┆ intriguin ┆ Prestige ┆ ┆ ┆ ┆ Blend ┆ │
│ ┆ ┆ g nose ┆ ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ with g... ┆ ┆ ┆ ┆ ┆ ┆ │
├╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┤
│ 150927 ┆ Italy ┆ This ┆ Terre di ┆ ... ┆ Fiano di ┆ null ┆ White ┆ Terredora │
│ ┆ ┆ classic ┆ Dora ┆ ┆ Avellino ┆ ┆ Blend ┆ │
│ ┆ ┆ example ┆ ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ comes ┆ ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ from ... ┆ ┆ ┆ ┆ ┆ ┆ │
├╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┤
│ 150928 ┆ France ┆ A perfect ┆ Grand ┆ ... ┆ Champagne ┆ null ┆ Champagne ┆ Gosset │
│ ┆ ┆ salmon ┆ Brut Rosé ┆ ┆ ┆ ┆ Blend ┆ │
│ ┆ ┆ shade, ┆ ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ with ┆ ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ sce... ┆ ┆ ┆ ┆ ┆ ┆ │
├╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┤
│ 150929 ┆ Italy ┆ More ┆ null ┆ ... ┆ Alto Adige ┆ null ┆ Pinot ┆ Alois │
│ ┆ ┆ Pinot ┆ ┆ ┆ ┆ ┆ Grigio ┆ Lageder │
│ ┆ ┆ Grigios ┆ ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ should ┆ ┆ ┆ ┆ ┆ ┆ │
│ ┆ ┆ taste ... ┆ ┆ ┆ ┆ ┆ ┆ │
└────────┴─────────┴───────────┴───────────┴─────┴────────────┴────────────┴───────────┴───────────┘
@r-brinkWe can see from the shapes that `fetch` catches 500 rows and `collect` retrieves all the rows.
Out of Core [NEW]
What if you dataset doesn’t fit in memory? This example is rather small, but in this day and age it is not unlikely that you are working on datsets that don’t fit in memory any more. Polars offers a very easy way to work with that.
Pretend that our dataset is not ~50MB, but 50GB. What can we do to, for example, Groupby country and do some calculations:
Not much changes, except that in collect() we add: "streaming=True"
Output
We have got the output that we are looking for. Polars offers several ways to output our analyses, even to other formats useful for further analyses (e.g. pandas dataframe (
to_pandas()) or numpy arrays (to_numpy()).
Final words from 2022
Polars is a new package that is gaining a lot of attention. At the time of writing this article, it has gathered more than 1300 stars on Github, which is impressive looking at the fact that is around for less than a year. It offers almost all the functions that we need to manipulate our dataframe. Next to that, it offers a lazy API that helps us optimising our queries before we execute them. Although we didn’t touch it is in this article, the benchmark of H20 shows that it is super efficient and fast. Especially with larger datasets it becomes worthwhile to look into the benefits that the lazy API has to offer.
This is what I wrote two years ago. 1300 stars. In 2022, while I am writing this, the project has collected 10.4k stars on Github and it is number 4 trending project on Github.
There are a lot of improvements happening under the hood and in the APIs.
For more information, check the Polars’ Github page. Here you can find links to benchmark’s, the Polars book or the Polars Discord.
I highly recommend joining the Discord. There is a lot of activity and many people are happy to help you with your specific questions.
