Real Python

Modeling Polymorphism in Django With Python

2019-01-02T14:00:00+00:00

Modeling polymorphism in relational databases is a challenging task. In this article, we present several modeling techniques to represent polymorphic objects in a relational database using the Django object-relational mapping (ORM).

This intermediate-level tutorial is designed for readers who are already familiar with the fundamental design of Django.

Free Bonus: Click here to get the most popular Django tutorials and resources on Real Python and improve your Django + Python web development skills.

What Is Polymorphism?

Polymorphism is the ability of an object to take on many forms. Common examples of polymorphic objects include event streams, different types of users, and products in an e-commerce website. A polymorphic model is used when a single entity requires different functionality or information.

In the examples above, all events are logged for future use, but they can contain different data. All users need be able to log in, but they might have different profile structures. In every e-commerce website, a user wants to put different products in their shopping cart.

Why Is Modeling Polymorphism Challenging?

There are many ways to model polymorphism. Some approaches use standard features of the Django ORM, and some use special features of the Django ORM. The main challenges you’ll encounter when modeling polymorphic objects are the following:

How to represent a single polymorphic object: Polymorphic objects have different attributes. The Django ORM maps attributes to columns in the database. In that case, how should the Django ORM map attributes to the columns in the table? Should different objects reside in the same table? Should you have multiple tables?
How to reference instances of a polymorphic model: To utilize database and Django ORM features, you need to reference objects using foreign keys. How you decide to represent a single polymorphic object is crucial to your ability to reference it.

To truly understand the challenges of modeling polymorphism, you are going to take a small bookstore from its first online website to a big online shop selling all sorts of products. Along the way, you’ll experience and analyze different approaches for modeling polymorphism using the Django ORM.

Note: To follow this tutorial, it is recommended that you use a PostgreSQL backend, Django 2.x, and Python 3.

It’s possible to follow along with other database backends as well. In places where features unique to PostgreSQL are used, an alternative will be presented for other databases.

Naive Implementation

You have a bookstore in a nice part of town right next to a coffee shop, and you want to start selling books online.

You sell only one type of product: books. In your online store, you want to show details about the books, like name and price. You want your users to browse around the website and collect many books, so you also need a cart. You eventually need to ship the books to the user, so you need to know the weight of each book to calculate the delivery fee.

Let’s create a simple model for your new book store:

from django.contrib.auth import get_user_model
from django.db import models


class Book(models.Model):
    name = models.CharField(
        max_length=100,
    )
    price = models.PositiveIntegerField(
        help_text='in cents',
    )
    weight = models.PositiveIntegerField(
        help_text='in grams',
    )

    def __str__(self) -> str:
        return self.name


class Cart(models.Model):
    user = models.OneToOneField(
        get_user_model(),
        primary_key=True,
        on_delete=models.CASCADE,
    )
    books = models.ManyToManyField(Book)

To create a new book, you provide a name, price, and weight:

>>>

>>> from naive.models import Book
>>> book = Book.objects.create(name='Python Tricks', price=1000, weight=200)
>>> book
<Product: Python Tricks>

To create a cart, you first need to associate it with a user:

>>>

>>> from django.contrib.auth import get_user_model
>>> haki = get_user_model().create_user('haki')

>>> from naive.models import Cart
>>> cart = Cart.objects.create(user=haki)

Then the user can start adding items to it:

>>>

>>> cart.products.add(book)
>>> cart.products.all()
<QuerySet [<Book: Python Tricks>]>

Pro

Easy to understand and maintain: It’s sufficient for a single type of product.

Con

Restricted to homogeneous products: It only supports products with the same set of attributes. Polymorphism is not captured or permitted at all.

Sparse Model

With the success of your online bookstore, users started to ask if you also sell e-books. E-books are a great product for your online store, and you want to start selling them right away.

A physical book is different from an e-book:

An e-book has no weight. It’s a virtual product.
An e-book does not require shipment. Users download it from the website.

To make your existing model support the additional information for selling e-books, you add some fields to the existing Book model:

from django.contrib.auth import get_user_model
from django.db import models


class Book(models.Model):
    TYPE_PHYSICAL = 'physical'
    TYPE_VIRTUAL = 'virtual'
    TYPE_CHOICES = (
        (TYPE_PHYSICAL, 'Physical'),
        (TYPE_VIRTUAL, 'Virtual'),
    )
    type = models.CharField(
        max_length=20,
        choices=TYPE_CHOICES,
    )

    # Common attributes
    name = models.CharField(
        max_length=100,
    )
    price = models.PositiveIntegerField(
        help_text='in cents',
    )

    # Specific attributes
    weight = models.PositiveIntegerField(
        help_text='in grams',
    )
    download_link = models.URLField(
        null=True, blank=True,
    )

    def __str__(self) -> str:
        return f'[{self.get_type_display()}] {self.name}'


class Cart(models.Model):
    user = models.OneToOneField(
        get_user_model(),
        primary_key=True,
        on_delete=models.CASCADE,
    )
    books = models.ManyToManyField(
        Book,
    )

First, you added a type field to indicate what type of book it is. Then, you added a URL field to store the download link of the e-book.

To add a physical book to your bookstore, do the following:

>>>

>>> from sparse.models import Book
>>> physical_book = Book.objects.create(
...     type=Book.TYPE_PHYSICAL,
...     name='Python Tricks',
...     price=1000,
...     weight=200,
...     download_link=None,
... )
>>> physical_book
<Book: [Physical] Python Tricks>

To add a new e-book, you do the following:

>>>

>>> virtual_book = Book.objects.create(
...     type=Book.TYPE_VIRTUAL,
...     name='The Old Man and the Sea',
...     price=1500,
...     weight=0,
...     download_link='https://books.com/12345',
... )
>>> virtual_book
<Book: [Virtual] The Old Man and the Sea>

Your users can now add both books and e-books to the cart:

>>>

>>> from sparse.models import Cart
>>> cart = Cart.objects.create(user=user)
>>> cart.books.add(physical_book, virtual_book)
>>> cart.books.all()
<QuerySet [<Book: [Physical] Python Tricks>, <Book: [Virtual] The Old Man and the Sea>]>

The virtual books are a big hit, and you decide to hire employees. The new employees are apparently not so tech savvy, and you start seeing weird things in the database:

>>>

>>> Book.objects.create(
...     type=Book.TYPE_PHYSICAL,
...     name='Python Tricks',
...     price=1000,
...     weight=0,
...     download_link='http://books.com/54321',
... )

That book apparently weighs 0 pounds and has a download link.

This e-book apparently weighs 100g and has no download link:

>>>

>>> Book.objects.create(
...     type=Book.TYPE_VIRTUAL,
...     name='Python Tricks',
...     price=1000,
...     weight=100,
...     download_link=None,
... )

This doesn’t make any sense. You have a data integrity problem.

To overcome integrity problems, you add validations to the model:

from django.core.exceptions import ValidationError


class Book(models.Model):

    # ...

    def clean(self) -> None:
        if self.type == Book.TYPE_VIRTUAL:
            if self.weight != 0:
                raise ValidationError(
                    'A virtual product weight cannot exceed zero.'
                )

            if self.download_link is None:
                raise ValidationError(
                    'A virtual product must have a download link.'
                )

        elif self.type == Book.TYPE_PHYSICAL:
            if self.weight == 0:
                raise ValidationError(
                    'A physical product weight must exceed zero.'
                )

            if self.download_link is not None:
                raise ValidationError(
                    'A physical product cannot have a download link.'
                )

        else:
            assert False, f'Unknown product type "{self.type}"'

You used Django’s built-in validation mechanism to enforce data integrity rules. clean() is only called automatically by Django forms. For objects that are not created by a Django form, you need to make sure to explicitly validate the object.

To keep the integrity of the Book model intact, you need to make a little change to the way you create books:

>>>

>>> book = Book(
...    type=Book.TYPE_PHYSICAL,
...    name='Python Tricks',
...    price=1000,
...    weight=0,
...    download_link='http://books.com/54321',
... )
>>> book.full_clean()
ValidationError: {'__all__': ['A physical product weight must exceed zero.']}

>>> book = Book(
...    type=Book.TYPE_VIRTUAL,
...    name='Python Tricks',
...    price=1000,
...    weight=100,
...    download_link=None,
... )
>>> book.full_clean()
ValidationError: {'__all__': ['A virtual product weight cannot exceed zero.']}

When creating objects using the default manager (Book.objects.create(...)), Django will create an object and immediately persist it to the database.

In your case, you want to validate the object before saving if to the database. You first create the object (Book(...)), validate it (book.full_clean()), and only then save it (book.save()).

Denormalization:

A sparse model is a product of denormalization. In a denormalization process, you inline attributes from multiple normalized models into a single table for better performance. A denormalized table will usually have a lot of nullable columns.

Denormalizing is often used in decision support systems such as data warehouses where read performance is most important. Unlike OLTP systems, data warehouses are usually not required to enforce data integrity rules, which makes denormalization ideal.

Pro

Easy to understand and maintain: The sparse model is usually the first step we take when certain types of objects need more information. It’s very intuitive and easy to understand.

Cons

Unable to utilize NOT NULL database constraints: Null values are used for attributes that are not defined for all types of objects.
Complex validation logic: Complex validation logic is required to enforce data integrity rules. The complex logic also requires more tests.
Many Null fields create clutter: Representing multiple types of products in a single model makes it harder to understand and maintain.
New types require schema changes: New types of products require additional fields and validations.

Use Case

The sparse model is ideal when you’re representing heterogeneous objects that share most attributes, and when new items are not added very often.

Semi-Structured Model

Your bookstore is now a huge success, and you are selling more and more books. You have books from different genres and publishers, e-books with different formats, books with odd shapes and sizes, and so on.

In the sparse model approach, you added fields for every new type of product. The model now has a lot of nullable fields, and new developers and employees are having trouble keeping up.

To address the clutter, you decide to keep only the common fields (name and price) on the model. You store the rest of the fields in a single JSONField:

from django.contrib.auth import get_user_model
from django.contrib.postgres.fields import JSONField
from django.db import models

class Book(models.Model):
    TYPE_PHYSICAL = 'physical'
    TYPE_VIRTUAL = 'virtual'
    TYPE_CHOICES = (
        (TYPE_PHYSICAL, 'Physical'),
        (TYPE_VIRTUAL, 'Virtual'),
    )
    type = models.CharField(
        max_length=20,
        choices=TYPE_CHOICES,
    )

    # Common attributes
    name = models.CharField(
        max_length=100,
    )
    price = models.PositiveIntegerField(
        help_text='in cents',
    )

    extra = JSONField()

    def __str__(self) -> str:
        return f'[{self.get_type_display()}] {self.name}'


class Cart(models.Model):
    user = models.OneToOneField(
        get_user_model(),
        primary_key=True,
        on_delete=models.CASCADE,
    )
    books = models.ManyToManyField(
        Book,
        related_name='+',
    )

JSONField:

In this example, you use PostgreSQL as a database backend. Django provides a built-in JSON field for PostgreSQL in django.contrib.postgres.fields.

For other databases, such as SQLite and MySQL, there are packages that provide similar functionality.

Your Book model is now clutter-free. Common attributes are modeled as fields. Attributes that are not common to all types of products are stored in the extra JSON field:

>>>

>>> from semi_structured.models import Book
>>> physical_book = Book(
...     type=Book.TYPE_PHYSICAL,
...     name='Python Tricks',
...     price=1000,
...     extra={'weight': 200},
... )
>>> physical_book.full_clean()
>>> physical_book.save()
<Book: [Physical] Python Tricks>

>>> virtual_book = Book(
...     type=Book.TYPE_VIRTUAL,
...     name='The Old Man and the Sea',
...     price=1500,
...     extra={'download_link': 'http://books.com/12345'},
... )
>>> virtual_book.full_clean()
>>> virtual_book.save()
<Book: [Virtual] The Old Man and the Sea>

>>> from semi_structured.models import Cart
>>> cart = Cart.objects.create(user=user)
>>> cart.books.add(physical_book, virtual_book)
>>> cart.books.all()
<QuerySet [<Book: [Physical] Python Tricks>, <Book: [Virtual] The Old Man and the Sea>]>

Clearing up the clutter is important, but it comes with a cost. The validation logic is a lot more complicated:

from django.core.exceptions import ValidationError
from django.core.validators import URLValidator

class Book(models.Model):

    # ...

    def clean(self) -> None:

        if self.type == Book.TYPE_VIRTUAL:

            try:
                weight = int(self.extra['weight'])
            except ValueError:
                raise ValidationError(
                    'Weight must be a number'
                )
            except KeyError:
                pass
            else:
                if weight != 0:
                    raise ValidationError(
                        'A virtual product weight cannot exceed zero.'
                    )

            try:
                download_link = self.extra['download_link']
            except KeyError:
                pass
            else:
                # Will raise a validation error
                URLValidator()(download_link)

        elif self.type == Book.TYPE_PHYSICAL:

            try:
                weight = int(self.extra['weight'])
            except ValueError:
                raise ValidationError(
                    'Weight must be a number'
                 )
            except KeyError:
                pass
            else:
                if weight == 0:
                    raise ValidationError(
                        'A physical product weight must exceed zero.'
                     )

            try:
                download_link = self.extra['download_link']
            except KeyError:
                pass
            else:
                if download_link is not None:
                    raise ValidationError(
                        'A physical product cannot have a download link.'
                    )

        else:
            raise ValidationError(f'Unknown product type "{self.type}"')

The benefit of using a proper field is that it validates the type. Both Django and the Django ORM can perform checks to make sure the right type is used for the field. When using a JSONField, you need to validate both the type and the value:

>>>

>>> book = Book.objects.create(
...     type=Book.TYPE_VIRTUAL,
...     name='Python Tricks',
...     price=1000,
...     extra={'weight': 100},
... )
>>> book.full_clean()
ValidationError: {'__all__': ['A virtual product weight cannot exceed zero.']}

Another issue with using JSON is that not all databases have proper support for querying and indexing values in JSON fields.

In PostgreSQL for example, you can query all the books that weigh more than 100:

>>>

>>> Book.objects.filter(extra__weight__gt=100)
<QuerySet [<Book: [Physical] Python Tricks>]>

However, not all database vendors support that.

Another restriction imposed when using JSON is that you are unable to use database constraints such as not null, unique, and foreign keys. You will have to implement these constraints in the application.

This semi-structured approach resembles NoSQL architecture and has many of its advantages and disadvantages. The JSON field is a way to get around the strict schema of a relational database. This hybrid approach provides us with the flexibility to squash many object types into a single table while still maintaining some of the benefits of a relational, strictly and strongly typed database. For many common NoSQL use cases, this approach might actually be more suitable.

Pros

Reduce clutter: Common fields are stored on the model. Other fields are stored in a single JSON field.
Easier to add new types: New types of products don’t require schema changes.

Cons

Complicated and ad hoc validation logic: Validating a JSON field requires validating types as well as values. This challenge can be addressed by using other solutions to validate JSON data such as JSON schema.
Unable to utilize database constraints: Database constraints such as null null, unique and foreign key constraints, which enforce type and data integrity at the database level, cannot be used.
Restricted by database support for JSON: Not all database vendors support querying and indexing JSON fields.
Schema is not enforced by the database system: Schema changes might require backward compatibility or ad hoc migrations. Data can “rot.”
No deep integration with the database metadata system: Metadata about the fields is not stored in the database. Schema is only enforced at the application level.

Use Case

A semi-structured model is ideal when you need to represent heterogeneous objects that don’t share many common attributes, and when new items are added often.

A classic use case for the semi-structured approach is storing events (like logs, analytics, and event stores). Most events have a timestamp, type and metadata like device, user agent, user, and so on. The data for each type is stored in a JSON field. For analytics and log events, it’s important to be able to add new types of events with minimal effort, so this approach is ideal.

Abstract Base Model

So far, you’ve worked around the problem of actually treating your products as heterogeneous. You worked under the assumption that the differences between the products is minimal, so it made sense to maintain them in the same model. This assumption can take you only so far.

Your little store is growing fast, and you want to start selling entirely different types of products, such as e-readers, pens, and notebooks.

A book and an e-book are both products. A product is defined using common attributes such as name and price. In an object-oriented environment, you could look at a Product as a base class or an interface. Every new type of product you add must implement the Product class and extend it with its own attributes.

Django offers the ability to create abstract base classes. Let’s define a Product abstract base class and add two models for Book and EBook:

from django.contrib.auth import get_user_model
from django.db import models


class Product(models.Model):
    class Meta:
        abstract = True

    name = models.CharField(
        max_length=100,
    )
    price = models.PositiveIntegerField(
        help_text='in cents',
    )

    def __str__(self) -> str:
        return self.name


class Book(Product):
    weight = models.PositiveIntegerField(
        help_text='in grams',
    )


class EBook(Product):
    download_link = models.URLField()

Notice that both Book and EBook inherit from Product. The fields defined in the base class Product are inherited, so the derived models Book and Ebook don’t need to repeat them.

To add new products, you use the derived classes:

>>>

>>> from abstract_base_model.models import Book
>>> book = Book.objects.create(name='Python Tricks', price=1000, weight=200)
>>> book
<Book: Python Tricks>

>>> ebook = EBook.objects.create(
...     name='The Old Man and the Sea',
...     price=1500,
...     download_link='http://books.com/12345',
... )
>>> ebook
<Book: The Old Man and the Sea>

You might have noticed that the Cart model is missing. You can try to create a Cart model with a ManyToMany field to Product:

class Cart(models.Model):
    user = models.OneToOneField(
       get_user_model(),
       primary_key=True,
       on_delete=models.CASCADE,
    )
    items = models.ManyToManyField(Product)

If you try to reference a ManyToMany field to an abstract model, you will get the following error:

abstract_base_model.Cart.items: (fields.E300) Field defines a relation with model 'Product', which is either not installed, or is abstract.

A foreign key constraint can only point to a concrete table. The abstract base model Product only exists in the code, so there is no products table in the database. The Django ORM will only create tables for the derived models Book and EBook.

Given that you can’t reference the abstract base class Product, you need to reference books and e-books directly:

class Cart(models.Model):
    user = models.OneToOneField(
        get_user_model(),
        primary_key=True,
        on_delete=models.CASCADE,
    )
    books = models.ManyToManyField(Book)
    ebooks = models.ManyToManyField(EBook)

You can now add both books and e-books to the cart:

>>>

>>> user = get_user_model().objects.first()
>>> cart = Cart.objects.create(user=user)
>>> cart.books.add(book)
>>> cart.ebooks.add(ebook)

This model is a bit more complicated now. Let’s query the total price of the items in the cart:

>>>

>>> from django.db.models import Sum
>>> from django.db.models.functions import Coalesce
>>> (
...     Cart.objects
...     .filter(pk=cart.pk)
...     .aggregate(total_price=Sum(
...         Coalesce('books__price', 'ebooks__price')
...     ))
... )
{'total_price': 1000}

Because you have more than one type of book, you use Coalesce to fetch either the price of the book or the price of the e-book for each row.

Pro

Easier to implement specific logic: A separate model for each product makes it easier to implement, test, and maintain specific logic.

Cons

Require multiple foreign keys: To reference all types of products, each type needs a foreign key.
Harder to implement and maintain: Operations on all types of products require checking all foreign keys. This adds complexity to the code and makes maintenance and testing harder.
Very hard to scale: New types of products require additional models. Managing many models can be tedious and very hard to scale.

Use Case

An abstract base model is a good choice when there are very few types of objects that required very distinct logic.

An intuitive example is modeling a payment process for your online shop. You want to accept payments with credit cards, PayPal, and store credit. Each payment method goes through a very different process that requires very distinct logic. Adding a new type of payment is not very common, and you don’t plan on adding new payment methods in the near future.

You create a payment process base class with derived classes for credit card payment process, PayPal payment process, and store credit payment process. For each of the derived classes, you implement the payment process in a very different way that cannot be easily shared. In this case, it might make sense to handle each payment process specifically.

Concrete Base Model

Django offers another way to implement inheritance in models. Instead of using an abstract base class that only exists in the code, you can make the base class concrete. “Concrete” means that the base class exists in the database as a table, unlike in the abstract base class solution, where the base class only exists in the code.

Using the abstract base model, you were unable to reference multiple type of products. You were forced to create a many-to-many relation for each type of product. This made it harder to perform tasks on the common fields such as getting the total price of all the items in the cart.

Using a concrete base class, Django will create a table in the database for the Product model. The Product model will have all the common fields you defined in the base model. Derived models such as Book and EBook will reference the Product table using a one-to-one field. To reference a product, you create a foreign key to the base model:

from django.contrib.auth import get_user_model
from django.db import models


class Product(models.Model):
    name = models.CharField(
        max_length=100,
    )
    price = models.PositiveIntegerField(
        help_text='in cents',
    )

    def __str__(self) -> str:
        return self.name


class Book(Product):
    weight = models.PositiveIntegerField()


class EBook(Product):
    download_link = models.URLField()

The only difference between this example and the previous one is that the Product model is not defined with abstract=True.

To create new products, you use derived Book and EBook models directly:

>>>

>>> from concrete_base_model.models import Book, EBook
>>> book = Book.objects.create(
...     name='Python Tricks',
...     price=1000,
...     weight=200,
... )
>>> book
<Book: Python Tricks>

>>> ebook = EBook.objects.create(
...     name='The Old Man and the Sea',
...     price=1500,
...     download_link='http://books.com/12345',
... )
>>> ebook
<Book: The Old Man and the Sea>

In the case of concrete base class, it’s interesting to see what’s happening in the underlying database. Let’s look at the tables created by Django in the database:

> \d concrete_base_model_product

Column |          Type          |                         Default
--------+-----------------------+---------------------------------------------------------
id     | integer                | nextval('concrete_base_model_product_id_seq'::regclass)
name   | character varying(100) |
price  | integer                |

Indexes:
   "concrete_base_model_product_pkey" PRIMARY KEY, btree (id)

Referenced by:
   TABLE "concrete_base_model_cart_items" CONSTRAINT "..." FOREIGN KEY (product_id) 
   REFERENCES concrete_base_model_product(id) DEFERRABLE INITIALLY DEFERRED

   TABLE "concrete_base_model_book" CONSTRAINT "..." FOREIGN KEY (product_ptr_id) 
   REFERENCES concrete_base_model_product(id) DEFERRABLE INITIALLY DEFERRED

   TABLE "concrete_base_model_ebook" CONSTRAINT "..." FOREIGN KEY (product_ptr_id) 
   REFERENCES concrete_base_model_product(id) DEFERRABLE INITIALLY DEFERRED

The product table has two familiar fields: name and price. These are the common fields you defined in the Product model. Django also created an ID primary key for you.

In the constraints section, you see multiple tables that are referencing the product table. Two tables that stand out are concrete_base_model_book and concrete_base_model_ebook:

> \d concrete_base_model_book

    Column     |  Type
---------------+---------
product_ptr_id | integer
weight         | integer

Indexes:
   "concrete_base_model_book_pkey" PRIMARY KEY, btree (product_ptr_id)

Foreign-key constraints:
   "..." FOREIGN KEY (product_ptr_id) REFERENCES concrete_base_model_product(id) 
   DEFERRABLE INITIALLY DEFERRED

The Book model has only two fields:

weight is the field you added in the derived Book model.
product_ptr_id is both the primary of the table and a foreign key to the base product model.

Behind the scenes, Django created a base table for product. Then, for each derived model, Django created another table that includes the additional fields, and a field that acts both as a primary key and a foreign key to the product table.

Let’s take a look at a query generated by Django to fetch a single book. Here are the results of print(Book.objects.filter(pk=1).query):

SELECT
    "concrete_base_model_product"."id",
    "concrete_base_model_product"."name",
    "concrete_base_model_product"."price",
    "concrete_base_model_book"."product_ptr_id",
    "concrete_base_model_book"."weight"
FROM
    "concrete_base_model_book"
    INNER JOIN "concrete_base_model_product" ON
        "concrete_base_model_book"."product_ptr_id" = "concrete_base_model_product"."id"
WHERE
    "concrete_base_model_book"."product_ptr_id" = 1

To fetch a single book, Django joined concrete_base_model_product and concrete_base_model_book on the product_ptr_id field. The name and price are in the product table and the weight is in the book table.

Since all the products are managed in the Product table, you can now reference it in a foreign key from the Cart model:

class Cart(models.Model):
    user = models.OneToOneField(
        get_user_model(),
        primary_key=True,
        on_delete=models.CASCADE,
    )
    items = models.ManyToManyField(Product)

Adding items to the cart is the same as before:

>>>

>>> from concrete_base_model.models import Cart
>>> cart = Cart.objects.create(user=user)
>>> cart.items.add(book, ebook)
>>> cart.items.all()
<QuerySet [<Book: Python Tricks>, <Book: The Old Man and the Sea>]>

Working with common fields is also simple:

>>>

>>> from django.db.models import Sum
>>> cart.items.aggregate(total_price=Sum('price'))
{'total_price': 2500}

Migrating base classes in Django:

When a derived model is created, Django adds a bases attribute to the migration:

  migrations.CreateModel(
      name='Book',
      fields=[...],
      bases=('concrete_base_model.product',),
  ),

If in the future you remove or change the base class, Django might not be able to perform the migration automatically. You might get this error:

TypeError: metaclass conflict: the metaclass of a derived class must 
be a (non-strict) subclass of the metaclasses of all its bases

This is a known issue in Django (#23818, #23521, #26488). To work around it, you must edit the original migration manually and adjust the bases attribute.

Pros

Primary key is consistent across all types: The product is issued by a single sequence in the base table. This restriction can be easily resolved by using a UUID instead of a sequence.
Common attributes can be queried from a single table: Common queries such as total price, list of product names, and prices can be fetched directly from the base table.

Cons

New product types require schema changes: A new type requires a new model.
Can produce inefficient queries: The data for a single item is in two database tables. Fetching a product requires a join with the base table.
Cannot access extended data from base class instance: A type field is required to downcast an item. This adds complexity to the code. django-polymorphic is a popular module that might eliminate some of these challenges.

Use Case

The concrete base model approach is useful when common fields in the base class are sufficient to satisfy most common queries.

For example, if you often need to query for the cart total price, show a list of items in the cart, or run ad hoc analytic queries on the cart model, you can benefit from having all the common attributes in a single database table.

Generic Foreign Key

Inheritance can sometimes be a nasty business. It forces you to create (possibly premature) abstractions, and it doesn’t always fit nicely into the ORM.

The main problem you have is referencing different products from the cart model. You first tried to squash all the product types into one model (sparse model, semi-structured model), and you got clutter. Then you tried splitting products into separate models and providing a unified interface using a concrete base model. You got a complicated schema and a lot of joins.

Django offers a special way of referencing any model in the project called GenericForeignKey. Generic foreign keys are part of the Content Types framework built into Django. The content type framework is used by Django itself to keep track of models. This is necessary for some core capabilities such as migrations and permissions.

To better understand what content types are and how they facilitate generic foreign keys, let’s look at the content type related to the Book model:

>>>

>>> from django.contrib.contenttypes.models import ContentType
>>> ct = ContentType.objects.get_for_model(Book)
>>> vars(ct)
{'_state': <django.db.models.base.ModelState at 0x7f1c9ea64400>,
'id': 22,
'app_label': 'concrete_base_model',
'model': 'book'}

Each model has a unique identifier. If you want to reference a book with PK 54, you can say, “Get object with PK 54 in the model represented by content type 22.”

GenericForeignKey is implemented exactly like that. To create a generic foreign key, you define two fields:

A reference to a content type (the model)
The primary key of the referenced object (the model instance’s pk attribute)

To implement a many-to-many relation using GenericForeignKey, you need to manually create a model to connect carts with items.

The Cart model remains roughly similar to what you have seen so far:

from django.db import models
from django.contrib.auth import get_user_model


class Cart(models.Model):
    user = models.OneToOneField(
        get_user_model(),
        primary_key=True,
        on_delete=models.CASCADE,
    )

Unlike previous Cart models, this Cart no longer includes a ManyToMany field. You are going need to do that yourself.

To represent a single item in the cart, you need to reference both the cart and any product:

from django.contrib.contenttypes.fields import GenericForeignKey
from django.contrib.contenttypes.models import ContentType


class CartItem(models.Model):
    cart = models.ForeignKey(
        Cart,
        on_delete=models.CASCADE,
        related_name='items',
    )
    product_object_id = models.IntegerField()
    product_content_type = models.ForeignKey(
        ContentType,
        on_delete=models.PROTECT,
    )
    product = GenericForeignKey(
        'product_content_type',
        'product_object_id',
    )

To add a new item in the Cart, you provide the content type and the primary key:

>>>

>>> book = Book.objects.first()

>>> Item.objects.create(
...     product_content_type=ContentType.objects.get_for_model(book),
...     product_object_id=book.pk,
... )
>>> ebook = EBook.objects.first()

>>> Item.objects.create(
...    product_content_type=ContentType.objects.get_for_model(ebook),
...    product_object_id=ebook.pk,
... )

Adding an item to a cart is a common task. You can add a method on the cart to add any product to the cart:

class Cart(models.Model):

    # ...

    def add_item(self, product) -> 'CartItem':
        product_content_type = ContentType.objects.get_for_model(product)

        return CartItem.objects.create(
            cart=self,
            product_content_type=product_content_type,
            product_object_id=product.pk,
        )

Adding a new item to a cart is now much shorter:

>>>

>>> cart.add_item(book)
>>> cart.add_item(ebook)

Getting information about the items in the cart is also possible:

>>>

>>> cart.items.all()
<QuerySet [<CartItem: CartItem object (1)>, <CartItem: CartItem object (2)>]

>>> item = cart.items.first()
>>> item.product
<Book: Python Tricks>

>>> item.product.price
1000

So far so good. Where’s the catch?

Let’s try to calculate the total price of the products in the cart:

>>>

>>> from django.db.models import Sum
>>> cart.items.aggregate(total=Sum('product__price'))

FieldError: Field 'product' does not generate an automatic reverse 
relation and therefore cannot be used for reverse querying. 
If it is a GenericForeignKey, consider adding a GenericRelation.

Django tells us it isn’t possible to traverse the generic relation from the generic model to the referenced model. The reason for that is that Django has no idea which table to join to. Remember, the Item model can point to any ContentType.

The error message does mention a GenericRelation. Using a GenericRelation, you can define a reverse relation from the referenced model to the Item model. For example, you can define a reverse relation from the Book model to items of books:

from django.contrib.contenttypes.fields import GenericRelation

class Book(model.Model):
    # ...
    cart_items = GenericRelation(
        'CartItem',
        'product_object_id',
        'product_content_type_id',
        related_query_name='books',
    )

Using the reverse relation, you can answer questions like how many carts include a specific book:

>>>

>>> book.cart_items.count()
4

>>> CartItem.objects.filter(books__id=book.id).count()
4

The two statement are identical.

You still need to know the price of the entire cart. You already saw that fetching the price from each product table is impossible using the ORM. To do that, you have to iterate the items, fetch each item separately, and aggregate:

>>>

>>> sum(item.product.price for item in cart.items.all())
2500

This is one of the major disadvantages of generic foreign keys. The flexibility comes with a great performance cost. It’s very hard to optimize for performance using just the Django ORM.

Structural Subtyping

In the abstract and concrete base class approaches, you used nominal subtyping, which is based on a class hierarchy. Mypy is able to detect this form of relation between two classes and infer types from it.

In the generic relation approach, you used structural subtyping. Structural subtyping exists when a class implements all the methods and attributes of another class. This form of subtyping is very useful when you wish to avoid direct dependency between modules.

Mypy provides a way to utilize structural subtyping using Protocols.

You already identified a product entity with common methods and attributes. You can define a Protocol:

from typing_extensions import Protocol

class Product(Protocol):
    pk: int
    name: str
    price: int

    def __str__(self) -> str:
        ...

Note: The use of class attributes and ellipses (...) in the method definition are new features in Python 3.7. In earlier versions of Python, it isn’t possible to define a Protocol using this syntax. Instead of an ellipsis, methods should have pass in the body. Class attributes such as pk and name can be defined using the @attribute decorator, but it will not work with Django models.

You can now use the Product protocol to add type information. For example, in add_item(), you accept an instance of a product and add it to the cart:

def add_item(
    self,
    product: Product,
) -> 'CartItem':
    product_content_type = ContentType.objects.get_for_model(product)

    return CartItem.objects.create(
        cart=self,
        product_content_type=product_content_type,
        product_object_id=product.pk,
    )

Running mypy on this function will not yield any warnings. Let’s say you change product.pk to product.id, which is not defined in the Product protocol:

def add_item(
    self,
    product: Product,
) -> 'CartItem':
    product_content_type = ContentType.objects.get_for_model(product)

    return CartItem.objects.create(
        cart=self,
        product_content_type=product_content_type,
        product_object_id=product.id,
    )

You will get the following warning from Mypy:

$ mypy
models.py:62: error: "Product" has no attribute "id"

Note: Protocol is not yet a part of Mypy. It’s part of the complementary package called mypy_extentions. The package is developed by the Mypy team and includes features that they thought weren’t ready for the main Mypy package yet.

Pros

Migrations are not needed to add product types: The generic foreign key can reference any model. Adding a new type of product does not require migrations.
Any model can be used as an item: Using generic foreign key, any model can be referenced by the Item model.
Built-in admin support: Django has built-in support for generic foreign keys in the admin. It can inline, for example, information about the referenced models in the detail page.
Self-contained module: There is no direct dependency between the products module and the cart module. This makes this approach ideal for existing projects and pluggable modules.

Cons

Can produce inefficient queries: The ORM cannot determine in advance what models are referenced by the generic foreign key. This makes it very difficult for it to optimize queries that fetch multiple types of products.
Harder to understand and maintain: Generic foreign key eliminates some Django ORM features that require access to specific product models. Accessing information from the product models requires writing more code.
Typing requires Protocol: Mypy is unable to provide type checking for generic models. A Protocol is required.

Use Case

Generic foreign keys are a great choice for pluggable modules or existing projects. The use of GenericForeignKey and structural subtyping abstract any direct dependency between the modules.

In the bookstore example, the book and e-book models can exist in a separate app and new products can be added without changing the cart module. For existing projects, a Cart module can be added with minimal changes to existing code.

The patterns presented in this article play nicely together. Using a mixture of patterns, you can eliminate some of the disadvantages and optimize the schema for your use case.

For example, in the generic foreign key approach, you were unable to get the price of the entire cart quickly. You had to fetch each item separately and aggregate. You can address this specific concern by inlining the price of the product on the Item model (the sparse model approach). This will allow you to query only the Item model to get the total price very quickly.

Conclusion

In this article, you started with a small town bookstore and grew it to a big e-commerce website. You tackled different types of problems and adjusted your model to accommodate the changes. You learned that problems such as complex code and difficulty adding new programmers to the team are often symptoms of a larger problem. You learned how to identify these problems and solve them.

You now know how to plan and implement a polymorphic model using the Django ORM. You’re familiar with multiple approaches, and you understand their pros and cons. You’re able to analyze your use case and decide on the best course of action.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python Community Interview With Corey Schafer

2018-12-31T14:00:00+00:00

For this week’s community interview, I am joined by Corey Schafer, of YouTube fame.

Corey is a full-time content creator publishing regular Python tutorials on YouTube. In this interview, we talk to Corey about his YouTube channel and his advice for budding YouTubers and content creators, getting his first developer job, and his passion for woodworking.

Ricky: Welcome to Real Python! We might as well start at the beginning. How’d you get into programming, and when did you start using Python?

Corey: Thanks for having me. I actually got started programming a little later in life than most people you’ve probably interviewed. I used to be a little self-conscious about getting started later, but now I try to be upfront about it so that others aren’t as intimidated if they are just getting started at an “older” age.

I suppose I technically got started in college when I began my degree in Computer Science, but even at that time, I wasn’t taking it very seriously. I would do enough to pass tests, but I wasn’t absorbing any of the information. I definitely wasn’t doing any side projects or using coding in any real-world applications.

It wasn’t until my mid-20s that I randomly applied to an internship for NASA at Kennedy Space Center and started to take programming seriously. To my surprise, I was chosen for the internship. Imposter syndrome set in full-force on day 1. I was definitely out of my league.

But the longer I worked there, the more I realized that these people weren’t superhuman. These were people just like me, with the exception that they’ve spent much more time putting in the work to master their skill.

I thought to myself, “If they can do it, there’s no reason I can’t.” So I left Cape Canaveral with a newfound motivation to really dive into programming and learn as much as I could. I really didn’t feel like I could call myself an actual programmer until my late 20s. I am in my early 30s now, so I feel like I haven’t even scratched the surface in terms of the skills I would like the master.

As far as learning Python, I didn’t start using it until about 4 years ago. I was a full-time front-end JavaScript developer at the West Virginia University GIS Technical Center doing some mapping work. We were using Python for some of the backend scripts, and I was assigned to maintain/update some of those. I found that I enjoyed that much more than the front-end JavaScript work and began using Python on a daily basis.

Ricky: You mentioned that you came to programming a little later in life (though still young). With that context in mind, what was your experience getting your first junior developer job, and do you have any advice for anyone looking for their first developer job later in life?

Corey: I got my first development job in my mid-20s, after my internship in Cape Canaveral. That job was at a small research company in West Virginia (WV). The competition isn’t as fierce in WV as it is in places like San Francisco or Silicon Valley, so I was able to land a position without much prior experience.

I was extremely intimidated when I first started there. I never wanted anyone to see any code that I had written out of fear that it would expose me as not knowing nearly as much as anyone else. I later came to find out that this is a common fear within the community.

Actually working as a developer full-time taught me much more than anything I had learned in school or self-study. Knowing the fundamentals is definitely helpful, but there’s no replacement for actually writing real-world applications and having your code critiqued by people who have been doing this for years.

It’s definitely uncomfortable for your mistakes to be on display for your coworkers to see, but once you get over that discomfort, you come out on the other side less likely to make the same mistakes in the future. Many of my viewers ask me, “How do you just know how to solve these problems?” Well, the truth is, many problems in programming are very similar.

Once you solve problems incorrectly over the years and are shown better and better ways, you eventually learn to recognize certain patterns and use the most efficient methods from the start. This isn’t a skill that most people have naturally. It is developed over years of trial and error.

Ricky: Most of our readers might recognize you from your very popular YouTube channel. I certainly heard of you through it. Not only is it on our Ultimate List of Python YouTube Channels, but I regularly watch your videos and even built one of my most recent Flask apps as a direct result of your Flask course. How have you found YouTube as a platform from a teaching point of view? Any surprising lessons learned?

Corey: Thanks. I was honored to have my channel featured in your article. YouTube is a terrific platform for learning, and I’m excited to see where online learning evolves in the future. Websites like YouTube have definitely lowered the barrier-to-entry for anyone who wants to create content.

Many people think you need a nice recording studio or some financial backing before you get started, but that is no longer the case. As long as you have a computer and a cell phone then you basically have all of the tools you need to get started. I have upgraded my equipment over time, but I first started on YouTube by doing screen recordings on a cheap laptop using the built-in laptop microphone.

For anyone who is thinking about starting a YouTube channel or creating content in general, I do have some lessons I’ve learned over time. I believe the most important lesson I’ve learned is that you should make content for yourself. I try not to create tutorials for topics for the sole purpose of them being popular or what will get the most views… I instead try to create the lessons that I wish I had when learning that topic.

Take notes while you learn certain subjects and keep track of what you found difficult to digest and why. If you get stuck on something, then most likely others are getting stuck on it as well. And once you find solutions to these problems, then you can look back and see if there is a way anyone could have explained it to you that would have helped you understand more easily. If there is, then be sure to pass that on to others.

I really believe that advice carries over into other fields. If you’re an educator, then make content that you personally would have found helpful. If you are a musician, then make music that you personally enjoy. If you’re a comedian, then tell jokes that you think are funny. If you do that, then most likely there are many people out there like yourself who will share your same mindset and love your content.

Ricky: What’s next for the YouTube channel going forward? Any plans to branch out into paid courses or other forms of teaching?

Corey: At the moment, I am sticking to YouTube courses. I feel very fortunate to have a platform that allows me to put my content out there for free and be funded by ad revenue. This allows people who can’t afford courses or who can’t pay for content at the moment to get access to all of my content.

Ideally, my content will always remain free as long as I have enough coming in to live on. It also wouldn’t be possible without generous supporters contributing through sites like Patreon and YouTube Channel Memberships. They support me financially on a monthly basis so that the content can remain free for those who can’t afford it. I would love to continue with this model into the future for as long as I can.

As far as branching out into other forms of teaching, I have thought about creating an online learning platform of some kind. I have used a lot of tools in my personal life that have helped me learn topics quickly. Tools such as spaced-repetition-learning applications and daily coding challenges.

I would love to tackle a project that brought these concepts to a learning platform in a way that would help students absorb material more quickly and fully. Any type of project like that will be something I wouldn’t work on for some time though. For now, I am focused on creating video content.

Ricky: You, sir, are a talented woodworker! Your stuff looks impressive. Is this a new hobby or a long-time passion of yours? How did you get started?

Corey: Oh, thank you for saying that. It is a hobby that I wish I had more time to explore. It is something I have been toying with for many years now. Personally, it is a fantastic way to clear my mind and drain any stress that’s built up.

Woodworking and programming can be similar in many ways. Sometimes I will start a woodworking project and have an idea of what I want the finished product to look like, but not a detailed understanding of where to begin. So you start by jotting down some outlines, then knock out the smaller components, and then after many hours of work you can combine all of that together into the result you were hoping for. There’s a lot of pride that comes with finishing those types of projects.

Ricky: Now for my last question. What else do you get up to in your spare time? What other hobbies and interests do you have, aside from Python? Any you’d like to share and/or plug?

Corey: Since I spend my professional career at a computer, I like to spend my spare time outside as much as possible. This includes hiking, kayaking, camping, swimming, and trips to the dog park. Now that I am working from home and have a more flexible schedule, my girlfriend and I would love to start traveling more. I believe I might mark some international PyCons on my calendar and use those as an excuse to visit some places we have wanted to see for a very long time, with the added bonus of meeting more folks from the Python community.

Thank you, Corey, for joining me this week. You can find Corey’s YouTube channel here. You can get in touch with him via Twitter or his website. And if you’ve benefited from Corey’s videos, consider supporting his efforts with via Patreon.

If there’s someone you’d like me to interview from the Python community, then reach out to me in the comments below, or send me a message on Twitter.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Make a Location-Based Web App With Django and GeoDjango

2018-12-24T14:00:00+00:00

Throughout this tutorial, you’ll learn how to use Django and GeoDjango to build a location-based web application from scratch. You’ll be building a simple nearby shops application that lists the shops closest to a user’s location.

By the end of this tutorial, you’ll be able to:

Use Django to build a simple web application from scratch
Use the GeoDjango sub-framework to implement geolocation features in your Django application
Use a spatial database (PostgreSQL and PostGIS) to get benefits from the spatial features and easily implement location-aware web apps

Free Bonus: Click here to get free access to additional Django tutorials and resources you can use to deepen your Python web development skills.

Example App Source Code: The full source code for the app you’ll be building in this tutorial is available on the realpython/materials repository on GitHub.

The Tools You Will Be Using

You’ll be using the following tools to develop your nearby shops web application:

The Python programming language
The Django web framework
The PostgreSQL database for persisting data
The PostGIS extension for supporting spatial features in the PostgreSQL database
pip for installing dependencies
The venv module for managing a virtual environment
Docker for installing PostgreSQL and PostGIS

Before diving into practical steps, let’s first start by introducing the frameworks you’ll be using.

Django is the most popular Python framework for building web apps. It makes it easy for developers to quickly build prototypes and meet their project deadlines by providing a plethora of built-in APIs and sub-frameworks such as GeoDjango.

GeoDjango is a built-in application that is included as a contrib module in Django. It’s actually a complete framework itself that can also be used separately from Django. It provides a toolbox of utilities for building GIS web applications.

GIS stands for Geographic Information System. It’s an information system (an organized system for the collection, organization, storage, and communication of information) designed for processing and manipulating data that have geographic or spatial characteristics.

GeoDjango also provides Python bindings to popular spatial libraries such as GEOS, GDAL, and GeoIP, which can be used separately without Django in any Python application or interactively in the shell.

GeoDjango aims to provide a world-class geographic web framework. It has been refactored over the years with the goal of making it easier to work with geospatial data, in other words data that identifies the geographic location of natural or artificial features on Earth and is stored as coordinates and topologies.

GeoDjango integrates very well with the Django ORM and provides a set of geometry fields defined by the Open Geospatial Consortium (OGS) that can be used to map to different types of geometries in geospatial databases:

GeometryField is the base class for all geometric fields in GeoDjango.
PointField is used for storing GEOS Point objects.
PolygonField is used for storing GEOS Polygon objects and so on.

GeoDjango is a very powerful framework for storing and working with geographic data using the Django ORM. It provides an easy-to-use API to find the distances between two points on a map, areas of polygons, the points within a polygon, and so on.

To be able to work with GeoDjango, you’ll need to have two things: a spatial database and geospatial libraries. A spatial database is a database that is optimized for storing and querying data that represents objects defined in a geometric space.

To fully use all features of GeoDjango, you’ll need to install the following open-source geospatial libraries:

GEOS stands for Geometry Engine Open Source. It’s a C++ port of the JTS (Java Topology Suite) that implements the OCG Simple Feature for SQL specification.
GDAL stands for Geospatial Data Abstraction Library. It’s an open-source library for working with raster and vector geospatial data formats.
PROJ.4 is for Cartographic Projections library. It’s an open-source GIS library for easily working with spatial reference systems and projections.
GeoIP is a library that helps users find geographical information based on an IP address.

This tutorial makes use of an Ubuntu 18.04 system for installing prerequisites and a Unix bash for running the commands, but this shouldn’t be a problem for you if you’re using any other system, particularly Unix-based systems like macOS.

For most installation instructions, you’ll be using the aptitude package manager, so you should simply replace that with the equivalent package manager for your system.

Prerequisites

In this section, you’ll be installing the prerequisites needed before you can bootstrap your project, such as Python 3 and GeoDjango dependencies (GEOS, GDAL, and PROJ.4). You’ll also use Docker to set up a PostgreSQL and PostGIS database for your project.

Installing Python 3

There is a big chance that you already have Python 3 installed on your system. If you don’t, you can simply head to the official website and download the binaries for your operating system.

Depending on your system, you may also be able to install Python 3 or upgrade it to the latest version if it’s already installed by using the official package manager.

If you have a problem installing Python 3 or want more information, you can check the Python 3 Installation & Setup Guide, which provides different ways to install Python 3 on your system.

Finally, you can check if you have Python 3 installed by running the following command:

$ python3 --version
Python 3.6.5

Installing GeoDjango Dependencies (GEOS, GDAL, and PROJ.4)

GeoDjango requires a spatial database and a set of open-source geospatial libraries:

GEOS is an open-source geometry engine and a C++ port of the JTS (Java Topology Suite). It’s required by GeoDjango for performing geometric operations.
PROJ.4 is an open-source GIS library for easily working with spatial reference systems and projections. You need it because you’ll be using PostGIS as the spatial database.
GDAL is an open-source geospatial data abstraction library for working with raster and vector data formats. It’s needed for many utilities used by GeoDjango.

You can refer to the docs for more information about spatial databases and the required libraries.

GDAL 2.2 is included in Ubuntu 18.04 so you can simply run the following command to install it:

$ sudo aptitude install gdal-bin libgdal-dev
$ sudo aptitude install python3-gdal

Note: python3-gdal is the Python 3 binding for GDAL.

Next, you can install the other libraries using the following:

$ sudo aptitude install binutils libproj-dev

Note: Since you’re using a binary package for GEOS, you also need to install binutils.

Refer to the docs for detailed instructions about how to install these dependencies on macOS and Windows.

For more information about PROJ.4, you can refer to its official docs.

Setting up a Spatial Database With PostgreSQL and PostGIS

You’ll use PostgreSQL, the most commonly used database with Django. It’s not a spatial database, but thanks to PostGIS you can power up your database with powerful geospatial features.

PostGIS is a spatial database extension that needs to be installed on a PostgreSQL database, which gives it the capability to store and work with spatial data and perform spatial operations. It adds support for geographic objects allowing location queries to be run in SQL.

You can either install PostgreSQL on your system, create a database, and then add the PostGIS extension, or better yet use Docker to quickly create a database using the kartoza postgis image, which provides a container with PostgreSQL and PostGIS already installed:

$ docker run --name=postgis -d -e POSTGRES_USER=user001 -e POSTGRES_PASS=123456789 -e POSTGRES_DBNAME=gis -p 5432:5432 kartoza/postgis:9.6-2.4

After running the command, you’ll have a PostgreSQL server listening on the 5432 port with a database called gis. The database uses the user001 username and the 123456789 password.

Note: You need to have Docker installed on your system. For instructions, you can simply refer to the official docs.

Setting up Your Project

Now that you have a spatial database set up and ready, you can go ahead and setup your Django project. In this section, you’ll use venv to create an isolated virtual environment for your project and install all the required packages such as Django.

Creating a Virtual Environment

A virtual environment allows you to create an isolated environment for the dependencies of your current project. This will allow you to avoid conflicts between the same packages that have different versions.

In Python 3, you can create virtual environments using virtualenv or the venv module.

For more information about Python virtual environments, check out Python Virtual Environments: A Primer.

Now, head over to your terminal and run the following command to create a virtual environment based on Python 3:

$ python3 -m venv env
(env) $

Next, you need to activate the following command:

$ source env/bin/activate

That’s it. You now have your virtual environment activated, and you can install the packages for your project.

Installing Django

The first step after creating and activating a virtual environment is to install Django. The Django package is available from the Python Package Index (PyPI) so you can simply use pip to install it by running the following command in your terminal:

$ pip install django

Creating a Django Project

The project you’ll be creating is a web application that lists shops sorted by distance so your users will be able to discover the shops that are close to their location.

The web application makes use of GeoDjango for easily implementing location requirements like calculating the distances of shops from the user’s location and ordering the shops by distance.

Using GeoDjango, you can get and display the nearest shops that are stored in a PostgreSQL database configured with the PostGIS extension to enable spatial operations.

Now, you’re ready to create a Django project using the django-admin.py script. Simply run the following command:

$ django-admin.py startproject nearbyshops

This will create a project named nearbyshops.

Configuring the PostgreSQL Database

Now that you’ve created a project, let’s continue by configuring the connection to the PostgreSQL and PostGIS spatial database. Open the settings.py file and add django.contrib.gis.db.backends.postgis as the engine with the credentials for the PostGIS database you configured earlier:

DATABASES = {
    'default': {
        'ENGINE': 'django.contrib.gis.db.backends.postgis',
        'NAME': 'gis',
        'USER': 'user001',
        'PASSWORD': '123456789',
        'HOST': 'localhost',
        'PORT': '5432'
    }
}

Note: You need to change the database credentials accordingly if you didn’t specify the same credentials when running the Docker container.

If you try to run your Django server at this point, you’ll get the ImportError: No module named 'psycopg2' error related to psycopg2, which is the most popular PostgreSQL adapter for Python. To solve the error, you simply need to install the psycopg2-binary in your virtual environment using the following:

$ pip install psycopg2-binary

Adding GeoDjango

GeoDjango is a framework that makes it as easy as possible to build GIS and location aware web applications. You can add it by simply including the gis contrib module in the list of installed apps.

Open the settings.py file and locate the INSTALLED_APPS array. Then add the 'django.contrib.gis' module:

INSTALLED_APPS = [
    # [...]
    'django.contrib.gis'
]

Creating a Django Application

A Django project is made up of applications. By default, it contains several core or built-in apps like django.contrib.admin, but you will usually add at least one app that contains your custom project’s code.

Note: For simple projects, you may only need one app, but once your project becomes bigger and has different requirements, you can organize your code in multiple separate apps.

Now that you have created a Django project, configured the connection with the spatial database, and added GeoDjango to the project, you need to create a Django application that you may call shops.

The shops application will contain the code for creating and displaying the shops closest to a user’s location. In the next steps, you are going to perform the following tasks:

Create the app
Add a Shop model
Add a data migration for loading initial demo data (shops)
Add a view function
Add a template

First run the following command to create the app:

$ python manage.py startapp shops

Next, you need to add it to the list of installed apps in the settings.py file, which will make Django recognize it as a part of your project:

INSTALLED_APPS = [
    # [...]
    'shops'
]

Creating a Django Model

After creating the shops application, which will contain the actual code of your project, you need to add models in your app. Django uses an ORM (Object Relational Mapper), which is an abstraction layer between Django and the database that transforms Python objects (or models) into database tables.

In this case, you need one model that represents a shop in the database. You’ll create a Shop model that has the following fields:

name: the name of the shop
location: the location of the shop in latitude and longitude coordinates
address: the address of the shop
city: the city the shop is in

Open the shops/models.py file and add the following code:

from django.contrib.gis.db import models

class Shop(models.Model):
    name = models.CharField(max_length=100)
    location = models.PointField()
    address = models.CharField(max_length=100)
    city = models.CharField(max_length=50)

For the location, you are using the PointField, a GeoDjango-specific geometric field for storing a GEOS Point object that represents a pair of longitude and latitude coordinates.

The other fields are normal Django fields of type CharField that can be used to store strings of small and large size.

Note: Please note that the models module is imported from django.contrib.gis.db and not the usual django.db module.

Creating the Database Tables

With Django, you don’t need to use SQL to create the database tables thanks to its ORM. Let’s create the database tables by using the makemigrations and migrate commands. Head back to your terminal and run the following:

$ python manage.py makemigrations
$ python manage.py migrate

For more information about these commands, check out Django Migrations – A Primer.

Adding a Super User

You need to create a super user so you can access the admin interface. This can be done using the following command:

$ python manage.py createsuperuser

The prompt will ask you for the username, email, and password you want to use for accessing the user account. Enter them and hit Enter.

Registering the Model in the Admin Interface

Django’s admin application provides a complete CRUD interface for managing data.

GeoDjango extends the admin application to add support for working with geometry fields.

Before you can access your models from Django admin, you need to register them.

Open the shops/admin.py file and add the following code:

from django.contrib.gis.admin import OSMGeoAdmin
from .models import Shop

@admin.register(Shop)
class ShopAdmin(OSMGeoAdmin):
    list_display = ('name', 'location')

You are using the @admin.register decorator to register the Shop model in the admin application. The decorated class is a representation of the Shop model in the admin interface and allows you to customize different aspects such as the Shop fields that you want to display. (In your case, it’s the name and location.) For more information about decorators, you can read Primer on Python Decorators.

Since the Shop model includes a GeoDjango field, you need to use the special OSMGeoAdmin class that’s available from the django.contrib.gis.admin package .

You can either use GeoModelAdmin or OSMGeoAdmin, which is a subclass of GeoModelAdmin that uses an Open Street Map layer in the admin to display geometric fields. This provides more information like street and thoroughfare details than would be available with the GeoModelAdmin class, which uses Vector Map Level 0.

You can now run the Django server:

$ python manage.py runserver

You application will be running from localhost:8000, and you can access the admin interface from localhost:8000/admin.

This is a screenshot from the Add shop interface:

You can see that the Location geometric field is displayed as an interactive map. You can zoom the map in and out, and you can choose different selectors at the top right corner of the map to select a location, which is marked by the green circle.

Adding Initial Data

You need some initial demo data for your application, but instead of manually adding data, you can use a data migration.

Data migrations can be used for multiple scenarios, including adding initial data in your database. For more information, check out Data Migrations.

Before creating a migration, let’s first get some real-world data from OpenStreetMap using overpass turbo, a web-based data filtering tool for OpenStreetMap. You can run Overpass API queries and analyze the resulting data interactively on the map.

You can also use the integrated Wizard, which makes it easy to create queries.

In your case, you want to get all the shops in a city. Simply click on the Wizard button. A small window will pop up. In the text field, write a query like “shop in Miami” and click on build and run query.

Next, click on the export button and click on download/copy as raw OSM data to download a JSON file that contains the raw OSM data. Save the file as data.json in your project’s root folder:

This is a screenshot of example data from the file:

You need to get the objects in the elements array. Specifically the lat, lon, and tags (name) fields for each shop.

You can find more details on how you can write Overpass queries from this wiki.

Now, let’s create an empty migration for importing the content of the data.json file in the database, using the following command:

$ python manage.py makemigrations shops --empty

Open the migration file. It has the following code:

from django.db import migrations

class Migration(migrations.Migration):
    dependencies = [
        ('shops', '0001_initial'),
    ]
    operations = [
    ]

You next need to create load_data() to be executed by RunPython(). First, in the import area, add the following imports:

from django.db import migrations
import json
from django.contrib.gis.geos import fromstr
from pathlib import Path

You are importing the Path class from the pathlib package for accessing low-level system functions, the json package for working with JSON, the Django built-in migrations API, and fromstr(), part of the geos package.

Next, add load_data():

DATA_FILENAME = 'data.json'
def load_data(apps, schema_editor):
    Shop = apps.get_model('shops', 'Shop')
    jsonfile = Path(__file__).parents[2] / DATA_FILENAME

    with open(str(jsonfile)) as datafile:
        objects = json.load(datafile)
        for obj in objects['elements']:
            try:
                objType = obj['type']
                if objType == 'node':
                    tags = obj['tags']
                    name = tags.get('name','no-name')
                    longitude = obj.get('lon', 0)
                    latitude = obj.get('lat', 0)
                    location = fromstr(f'POINT({longitude} {latitude})', srid=4326)
                    Shop(name=name, location = location).save()
            except KeyError:
                pass

Let’s explain the code you’ve just added. You first construct the absolute path using the Path class of the pathlib library and open the data.json file. Next, you parse the JSON file into a Python object.

You loop through the elements object containing the locations and tags of shops. Inside the loop, you extract the name and the longitude and latitude coordinates. Then you use formstr() to return a valid GEOSGeometry object corresponding to the spatial data in the string that can be assigned to the location field of the Shop model. Finally, you create and save the instance of the Shop model corresponding to the extracted data.

You are using the with statement, so you don’t have to explicitly close the file, and an f-string for formatting the argument of fromstr().

fromstr() takes a srid as the second parameter. srid stands for Spatial Reference System Identifier. It’s a unique value to identify spatial reference systems (projection systems used for interpreting the data in the spatial database).

The 4326 srid is the most popular system used with PostGIS. It’s also known as WGS84, where units are specified in degrees of longitude and latitude. You can refer to spatialreference.org for a Django-powered database of spatial reference systems.

Next, add the migration class to execute the above function when you run the migrate command:

class Migration(migrations.Migration):

    dependencies = [
        ('shops', '0005_auto_20181018_2050'),
    ]

    operations = [
        migrations.RunPython(load_data)
    ]

That’s it. You can now return to your terminal and run the following:

$ python manage.py migrate

The data from the data.json file will be loaded on your database. Run your Django server and head to your admin interface. You should see your data in the table. In my case, this is a screenshot of a portion of the table:

Displaying Nearby Shops

At this point of the tutorial, you have created:

The shops application, which encapsulates the code for creating and getting nearby shops in your project
The Shop model and the corresponding tables in the database
The initial admin user for accessing the admin interface
The initial demo data for loading real-world shops in the database that you can play with without manually entering a lot of fake data

You also registered the Shop model in the admin application so you can create, update, delete, and list shops from the admin interface.

Next, you’ll add a view function using the generic ListView class that you can use to display a list of nearby shops. You also create an HTML template that will be used by the view function to render the shops and add the URL that will be used to display the shops.

Let’s start by adding a template and a view function that will be used to display the nearby shops from a user’s location.

Open the shops/views.py file and start by importing the necessary APIs:

from django.views import generic
from django.contrib.gis.geos import fromstr
from django.contrib.gis.db.models.functions import Distance
from .models import Shop

Next add a user_location variable where you can hard code a user location:

longitude = -80.191788
latitude = 25.761681

user_location = Point(longitude, latitude, srid=4326)

In this part, you’ll simply hard code the user’s location (the coordinates of Miami in the USA), but this ideally should be specified by the user or retrieved automatically from the user’s browser with their permission using JavaScript and the HTML5 GeoLocation API. You can scroll down to the middle of that page to see a live example of implementing the Geolocation API.

Finally, add the following view class:

class Home(generic.ListView):
    model = Shop
    context_object_name = 'shops'
    queryset = Shop.objects.annotate(distance=Distance('location',
    user_location)
    ).order_by('distance')[0:6]
    template_name = 'shops/index.html'

You are using the generic class-based ListView to create a view.

Class-based views are an alternative way to implement views as Python classes instead of functions. They are used to handle common use cases in web development without re-inventing the wheel. In this example, you’ve just sub-classed the ListView generic view and overridden the model, context_object_name, queryset, and template_name attributes to create a list view that handles HTTP requests without any extra code.

Now let’s focus on the queryset attribute. To get the nearby shops, you simply use .annotate() to annotate each object on the returned queryset with a distance annotation that’s calculated using Distance(), available from GeoDjango, between the location of each shop and the user’s location. You also order the returned queryset by the distance annotation and take only the nearest six shops.

You can learn more about class-based views from the official docs.

Next let’s add the shops/index.html template with the following content:

<!DOCTYPE html>
<html lang="en">
    <head>
        <meta charset="utf-8">
        <title>Nearby Shops</title>
    </head>
<body>
    <h1>Nearby Shops</h1>
    {% if shops %}
    <ul>
    {% for shop in shops %}
        <li>
        {{ shop.name }}: {{shop.distance}}
        </li>
    {% endfor %}
    </ul>
    {% endif %}
</body>
</html>

The nearest shops are available from the shops context object that you specified as the context_object_name in the class-based view. You loop through the shops object and you display the name and distance from the user’s location for each shop.

Finally, let’s add a URL to our urls.py file:

from django.urls import path
from shops import views

urlpatterns = [
    # [...]
    path('', views.ShopList.as_view())
]

You use .as_view() to return a callable view that takes a request and returns a response, which can be passed as the second parameter for path() that maps paths to views.

Now you can run your Django server. The home page will display a simple un-styled list with the nearest shops from the hard-coded user’s location. This is an example screenshot:

In the screenshot, each item in the list displays the name of the shop (before the colon) and the distance in meters from the user’s location (after the colon). The letter m refers to meters.

Note: The view function you used to display the results in the screenshot is just for testing the queryset annotated by Distance().

In the next part of the series, you’ll use a RESTful endpoint to return the nearest shops. Instead of a Django template, you’ll use Vue.js with some CSS styling for a better look, so stay tuned!

Wrap-Up

Congratulations on creating your location based web application using GeoDjango, which aims to become a world-class geographic framework for implementing GIS apps. You now have the basic skills that you can use to either add simple geolocation stuff to your applications or create GIS apps. You can read the GeoDjango docs for a complete resource of the available APIs and what you can do with them.

You also learned to use Docker to quickly pull and launch a PostgreSQL and PostGIS server. Docker can be used for more than that. It’s a containerization tool for spinning up isolated, reproducible application environments. You can read Django Development with Docker Compose and Machine if you want to learn how to containerize your Django project.

Nowadays, location aware apps (apps that know your location and help you discover nearby objects and services by offering you results based on your location) are all the rage. Using the knowledge you gained in this tutorial, you’ll be able to incorporate this modern feature in your apps developed with Django.

The only requirement, besides the dependencies of GeoDjango, is to use a spatial database (a database that’s capable of storing and manipulating spatial data). For PostgreSQL, one of the most popular database management systems used with Django, you can simply install the PostGIS extension with your database to turn it into a spatial database. Other popular databases like Oracle and MySQL have built-in support for spatial data.

Conclusion

Throughout this tutorial, you used PostgreSQL, PostGIS, Django, and GeoDjango to build a simple nearby shops web application.

Example App Source Code: The full source code for the app you’ll be building in this tutorial is available on the realpython/materials repository on GitHub.

Free Bonus: Click here to get free access to additional Django tutorials and resources you can use to deepen your Python web development skills.

In the next tutorial, you’ll continue building the application with Django REST framework to expose a RESTful API, and you’ll use the JavaScript Vue.js library with the Axios HTTP client to consume the API and render the data.

You’ll also use the HTML5 Geolocation API to automatically get the user’s location from the user’s browser instead of hard-coding it like you did in this tutorial, so stay tuned!

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

How to Write Beautiful Python Code With PEP 8

2018-12-19T14:00:00+00:00

PEP 8, sometimes spelled PEP8 or PEP-8, is a document that provides guidelines and best practices on how to write Python code. It was written in 2001 by Guido van Rossum, Barry Warsaw, and Nick Coghlan. The primary focus of PEP 8 is to improve the readability and consistency of Python code.

PEP stands for Python Enhancement Proposal, and there are several of them. A PEP is a document that describes new features proposed for Python and documents aspects of Python, like design and style, for the community.

This tutorial outlines the key guidelines laid out in PEP 8. It’s aimed at beginner to intermediate programmers, and as such I have not covered some of the most advanced topics. You can learn about these by reading the full PEP 8 documentation.

By the end of this tutorial, you’ll be able to:

Write Python code that conforms to PEP 8
Understand the reasoning behind the guidelines laid out in PEP 8
Set up your development environment so that you can start writing PEP 8 compliant Python code

Free Bonus: 5 Thoughts On Python Mastery, a free course for Python developers that shows you the roadmap and the mindset you'll need to take your Python skills to the next level.

Why We Need PEP 8

“Readability counts.”

— The Zen of Python

PEP 8 exists to improve the readability of Python code. But why is readability so important? Why is writing readable code one of the guiding principles of the Python language?

As Guido van Rossum said, “Code is read much more often than it is written.” You may spend a few minutes, or a whole day, writing a piece of code to process user authentication. Once you’ve written it, you’re never going to write it again. But you’ll definitely have to read it again. That piece of code might remain part of a project you’re working on. Every time you go back to that file, you’ll have to remember what that code does and why you wrote it, so readability matters.

If you’re new to Python, it can be difficult to remember what a piece of code does a few days, or weeks, after you wrote it. If you follow PEP 8, you can be sure that you’ve named your variables well. You’ll know that you’ve added enough whitespace so it’s easier to follow logical steps in your code. You’ll also have commented your code well. All this will mean your code is more readable and easier to come back to. As a beginner, following the rules of PEP 8 can make learning Python a much more pleasant task.

Following PEP 8 is particularly important if you’re looking for a development job. Writing clear, readable code shows professionalism. It’ll tell an employer that you understand how to structure your code well.

If you have more experience writing Python code, then you may need to collaborate with others. Writing readable code here is crucial. Other people, who may have never met you or seen your coding style before, will have to read and understand your code. Having guidelines that you follow and recognize will make it easier for others to read your code.

Naming Conventions

“Explicit is better than implicit.”

— The Zen of Python

When you write Python code, you have to name a lot of things: variables, functions, classes, packages, and so on. Choosing sensible names will save you time and energy later. You’ll be able to figure out, from the name, what a certain variable, function, or class represents. You’ll also avoid using inappropriate names that might result in errors that are difficult to debug.

Note: Never use l, O, or I single letter names as these can be mistaken for 1 and 0, depending on typeface:

O = 2  # This may look like you're trying to reassign 2 to zero

Naming Styles

The table below outlines some of the common naming styles in Python code and when you should use them:

Type	Naming Convention	Examples
Function	Use a lowercase word or words. Separate words by underscores to improve readability.	`function`, `my_function`
Variable	Use a lowercase single letter, word, or words. Separate words with underscores to improve readability.	`x`, `var`, `my_variable`
Class	Start each word with a capital letter. Do not separate words with underscores. This style is called camel case.	`Model`, `MyClass`
Method	Use a lowercase word or words. Separate words with underscores to improve readability.	`class_method`, `method`
Constant	Use an uppercase single letter, word, or words. Separate words with underscores to improve readability.	`CONSTANT`, `MY_CONSTANT`, `MY_LONG_CONSTANT`
Module	Use a short, lowercase word or words. Separate words with underscores to improve readability.	`module.py`, `my_module.py`
Package	Use a short, lowercase word or words. Do not separate words with underscores.	`package`, `mypackage`

These are some of the common naming conventions and examples of how to use them. But in order to write readable code, you still have to be careful with your choice of letters and words. In addition to choosing the correct naming styles in your code, you also have to choose the names carefully. Below are a few pointers on how to do this as effectively as possible.

How to Choose Names

Choosing names for your variables, functions, classes, and so forth can be challenging. You should put a fair amount of thought into your naming choices when writing code as it will make your code more readable. The best way to name your objects in Python is to use descriptive names to make it clear what the object represents.

When naming variables, you may be tempted to choose simple, single-letter lowercase names, like x. But, unless you’re using x as the argument of a mathematical function, it’s not clear what x represents. Imagine you are storing a person’s name as a string, and you want to use string slicing to format their name differently. You could end up with something like this:

>>>

>>> # Not recommended
>>> x = 'John Smith'
>>> y, z = x.split()
>>> print(z, y, sep=', ')
'Smith, John'

This will work, but you’ll have to keep track of what x, y, and z represent. It may also be confusing for collaborators. A much clearer choice of names would be something like this:

>>>

>>> # Recommended
>>> name = 'John Smith'
>>> first_name, last_name = name.split()
>>> print(last_name, first_name, sep=', ')
'Smith, John'

Similarly, to reduce the amount of typing you do, it can be tempting to use abbreviations when choosing names. In the example below, I have defined a function db() that takes a single argument x and doubles it:

# Not recommended
def db(x):
    return x * 2

At first glance, this could seem like a sensible choice. db() could easily be an abbreviation for double. But imagine coming back to this code in a few days. You may have forgotten what you were trying to achieve with this function, and that would make guessing how you abbreviated it difficult.

The following example is much clearer. If you come back to this code a couple of days after writing it, you’ll still be able to read and understand the purpose of this function:

# Recommended
def multiply_by_two(x):
    return x * 2

The same philosophy applies to all other data types and objects in Python. Always try to use the most concise but descriptive names possible.

Code Layout

“Beautiful is better than ugly.”

— The Zen of Python

How you lay out your code has a huge role in how readable it is. In this section, you’ll learn how to add vertical whitespace to improve the readability of your code. You’ll also learn how to handle the 79 character line limit recommended in PEP 8.

Blank Lines

Vertical whitespace, or blank lines, can greatly improve the readability of your code. Code that’s bunched up together can be overwhelming and hard to read. Similarly, too many blank lines in your code makes it look very sparse, and the reader might need to scroll more than necessary. Below are three key guidelines on how to use vertical whitespace.

Surround top-level functions and classes with two blank lines. Top-level functions and classes should be fairly self-contained and handle separate functionality. It makes sense to put extra vertical space around them, so that it’s clear they are separate:

class MyFirstClass:
    pass


class MySecondClass:
    pass


def top_level_function():
    return None

Surround method definitions inside classes with a single blank line. Inside a class, functions are all related to one another. It’s good practice to leave only a single line between them:

class MyClass:
    def first_method(self):
        return None

    def second_method(self):
        return None

Use blank lines sparingly inside functions to show clear steps. Sometimes, a complicated function has to complete several steps before the return statement. To help the reader understand the logic inside the function, it can be helpful to leave a blank line between each step.

In the example below, there is a function to calculate the variance of a list. This is two-step problem, so I have indicated each step by leaving a blank line between them. There is also a blank line before the return statement. This helps the reader clearly see what’s returned:

def calculate_variance(number_list):
    sum_list = 0
    for number in number_list:
        sum_list = sum_list + number
    mean = sum_list / len(number_list)

    sum_squares = 0
    for number in number_list:
        sum_squares = sum_squares + number**2
    mean_squares = sum_squares / len(number_list)

    return mean_squares - mean**2

If you use vertical whitespace carefully, it can greatly improved the readability of your code. It helps the reader visually understand how your code splits up into sections, and how those sections relate to one another.

Maximum Line Length and Line Breaking

PEP 8 suggests lines should be limited to 79 characters. This is because it allows you to have multiple files open next to one another, while also avoiding line wrapping.

Of course, keeping statements to 79 characters or less is not always possible. PEP 8 outlines ways to allow statements to run over several lines.

Python will assume line continuation if code is contained within parentheses, brackets, or braces:

def function(arg_one, arg_two,
             arg_three, arg_four):
    return arg_one

If it is impossible to use implied continuation, then you can use backslashes to break lines instead:

from mypkg import example1, \
    example2, example3

However, if you can use implied continuation, then you should do so.

If line breaking needs to occur around binary operators, like + and *, it should occur before the operator. This rule stems from mathematics. Mathematicians agree that breaking before binary operators improves readability. Compare the following two examples.

Below is an example of breaking before a binary operator:

# Recommended
total = (first_variable
         + second_variable
         - third_variable)

You can immediately see which variable is being added or subtracted, as the operator is right next to the variable being operated on.

Now, let’s see an example of breaking after a binary operator:

# Not Recommended
total = (first_variable +
         second_variable -
         third_variable)

Here, it’s harder to see which variable is being added and which is subtracted.

Breaking before binary operators produces more readable code, so PEP 8 encourages it. Code that consistently breaks after a binary operator is still PEP 8 compliant. However, you’re encouraged to break before a binary operator.

Indentation

“There should be one—and preferably only one—obvious way to do it.”

— The Zen of Python

Indentation, or leading whitespace, is extremely important in Python. The indentation level of lines of code in Python determines how statements are grouped together.

Consider the following example:

x = 3
if x > 5:
    print('x is larger than 5')

The indented print statement lets Python know that it should only be executed if the if statement returns True. The same indentation applies to tell Python what code to execute when a function is called or what code belongs to a given class.

The key indentation rules laid out by PEP 8 are the following:

Use 4 consecutive spaces to indicate indentation.
Prefer spaces over tabs.

Tabs vs. Spaces

As mentioned above, you should use spaces instead of tabs when indenting code. You can adjust the settings in your text editor to output 4 spaces instead of a tab character, when you press the Tab key.

If you’re using Python 2 and have used a mixture of tabs and spaces to indent your code, you won’t see errors when trying to run it. To help you to check consistency, you can add a -t flag when running Python 2 code from the command line. The interpreter will issue warnings when you are inconsistent with your use of tabs and spaces:

$ python2 -t code.py
code.py: inconsistent use of tabs and spaces in indentation

If, instead, you use the -tt flag, the interpreter will issue errors instead of warnings, and your code will not run. The benefit of using this method is that the interpreter tells you where the inconsistencies are:

$ python2 -tt code.py
  File "code.py", line 3
    print(i, j)
             ^
TabError: inconsistent use of tabs and spaces in indentation

Python 3 does not allow mixing of tabs and spaces. Therefore, if you are using Python 3, then these errors are issued automatically:

$ python3 code.py
  File "code.py", line 3
    print(i, j)
              ^
TabError: inconsistent use of tabs and spaces in indentation

You can write Python code with either tabs or spaces indicating indentation. But, if you’re using Python 3, you must be consistent with your choice. Otherwise, your code will not run. PEP 8 recommends that you always use 4 consecutive spaces to indicate indentation.

Indentation Following Line Breaks

When you’re using line continuations to keep lines to under 79 characters, it is useful to use indentation to improve readability. It allows the reader to distinguish between two lines of code and a single line of code that spans two lines. There are two styles of indentation you can use.

The first of these is to align the indented block with the opening delimiter:

def function(arg_one, arg_two,
             arg_three, arg_four):
    return arg_one

Sometimes you can find that only 4 spaces are needed to align with the opening delimiter. This will often occur in if statements that span multiple lines as the if, space, and opening bracket make up 4 characters. In this case, it can be difficult to determine where the nested code block inside the if statement begins:

x = 5
if (x > 3 and
    x < 10):
    print(x)

In this case, PEP 8 provides two alternatives to help improve readability:

Add a comment after the final condition. Due to syntax highlighting in most editors, this will separate the conditions from the nested code:
```
x = 5
if (x > 3 and
    x < 10):
    # Both conditions satisfied
    print(x)
```

Add extra indentation on the line continuation:

x = 5
if (x > 3 and
        x < 10):
    print(x)

An alternative style of indentation following a line break is a hanging indent. This is a typographical term meaning that every line but the first in a paragraph or statement is indented. You can use a hanging indent to visually represent a continuation of a line of code. Here’s an example:

var = function(
    arg_one, arg_two,
    arg_three, arg_four)

Note: When you’re using a hanging indent, there must not be any arguments on the first line. The following example is not PEP 8 compliant:

# Not Recommended
var = function(arg_one, arg_two,
    arg_three, arg_four)

When using a hanging indent, add extra indentation to distinguish the continued line from code contained inside the function. The following example is difficult to read because the code inside the function is at the same indentation level as the continued lines:

# Not Recommended
def function(
    arg_one, arg_two,
    arg_three, arg_four):
    return arg_one

Instead, it’s better to use a double indent on the line continuation. This helps you to distinguish between function arguments and the function body, improving readability:

def function(
        arg_one, arg_two,
        arg_three, arg_four):
    return arg_one

When you write PEP 8 compliant code, the 79 character line limit forces you to add line breaks in your code. To improve readability, you should indent a continued line to show that it is a continued line. There are two ways of doing this. The first is to align the indented block with the opening delimiter. The second is to use a hanging indent. You are free to chose which method of indentation you use following a line break.

Where to Put the Closing Brace

Line continuations allow you to break lines inside parentheses, brackets, or braces. It’s easy to forget about the closing brace, but it’s important to put it somewhere sensible. Otherwise, it can confuse the reader. PEP 8 provides two options for the position of the closing brace in implied line continuations:

Line up the closing brace with the first non-whitespace character of the previous line:

list_of_numbers = [
    1, 2, 3,
    4, 5, 6,
    7, 8, 9
    ]

Line up the closing brace with the first character of the line that starts the construct:
```
list_of_numbers = [
    1, 2, 3,
    4, 5, 6,
    7, 8, 9
]
```

You are free to chose which option you use. But, as always, consistency is key, so try to stick to one of the above methods.

Comments

“If the implementation is hard to explain, it’s a bad idea.”

— The Zen of Python

You should use comments to document code as it’s written. It is important to document your code so that you, and any collaborators, can understand it. When you or someone else reads a comment, they should be able to easily understand the code the comment applies to and how it fits in with the rest of your code.

Here are some key points to remember when adding comments to your code:

Limit the line length of comments and docstrings to 72 characters.
Use complete sentences, starting with a capital letter.
Make sure to update comments if you change your code.

Block Comments

Use block comments to document a small section of code. They are useful when you have to write several lines of code to perform a single action, such as importing data from a file or updating a database entry. They are important as they help others understand the purpose and functionality of a given code block.

PEP 8 provides the following rules for writing block comments:

Indent block comments to the same level as the code they describe.
Start each line with a # followed by a single space.
Separate paragraphs by a line containing a single #.

Here is a block comment explaining the function of a for loop. Note that the sentence wraps to a new line to preserve the 79 character line limit:

for i in range(0, 10):
    # Loop over i ten times and print out the value of i, followed by a
    # new line character
    print(i, '\n')

Sometimes, if the code is very technical, then it is necessary to use more than one paragraph in a block comment:

def quadratic(a, b, c, x):
    # Calculate the solution to a quadratic equation using the quadratic
    # formula.
    #
    # There are always two solutions to a quadratic equation, x_1 and x_2.
    x_1 = (- b+(b**2-4*a*c)**(1/2)) / (2*a)
    x_2 = (- b-(b**2-4*a*c)**(1/2)) / (2*a)
    return x_1, x_2

If you’re ever in doubt as to what comment type is suitable, then block comments are often the way to go. Use them as much as possible throughout your code, but make sure to update them if you make changes to your code!

Inline Comments

Inline comments explain a single statement in a piece of code. They are useful to remind you, or explain to others, why a certain line of code is necessary. Here’s what PEP 8 has to say about them:

Use inline comments sparingly.
Write inline comments on the same line as the statement they refer to.
Separate inline comments by two or more spaces from the statement.
Start inline comments with a # and a single space, like block comments.
Don’t use them to explain the obvious.

Below is an example of an inline comment:

x = 5  # This is an inline comment

Sometimes, inline comments can seem necessary, but you can use better naming conventions instead. Here’s an example:

x = 'John Smith'  # Student Name

Here, the inline comment does give extra information. However using x as a variable name for a person’s name is bad practice. There’s no need for the inline comment if you rename your variable:

student_name = 'John Smith'

Finally, inline comments such as these are bad practice as they state the obvious and clutter code:

empty_list = []  # Initialize empty list

x = 5
x = x * 5  # Multiply x by 5

Inline comments are more specific than block comments, and it’s easy to add them when they’re not necessary, which leads to clutter. You could get away with only using block comments so, unless you are sure you need an inline comment, your code is more likely to be PEP 8 compliant if you stick to block comments.

Documentation Strings

Documentation strings, or docstrings, are strings enclosed in double (""") or single (''') quotation marks that appear on the first line of any function, class, method, or module. You can use them to explain and document a specific block of code. There is an entire PEP, PEP 257, that covers docstrings, but you’ll get a summary in this section.

The most important rules applying to docstrings are the following:

Surround docstrings with three double quotes on either side, as in """This is a docstring""".
Write them for all public modules, functions, classes, and methods.

Put the """ that ends a multiline docstring on a line by itself:

def quadratic(a, b, c, x):
    """Solve quadratic equation via the quadratic formula.

    A quadratic equation has the following form:
    ax**2 + bx + c = 0

    There always two solutions to a quadratic equation: x_1 & x_2.
    """
    x_1 = (- b+(b**2-4*a*c)**(1/2)) / (2*a)
    x_2 = (- b-(b**2-4*a*c)**(1/2)) / (2*a)

    return x_1, x_2

For one-line docstrings, keep the """ on the same line:

def quadratic(a, b, c, x):
    """Use the quadratic formula"""
    x_1 = (- b+(b**2-4*a*c)**(1/2)) / (2*a)
    x_2 = (- b-(b**2-4*a*c)**(1/2)) / (2*a)

    return x_1, x_2

For a more detailed article on documenting Python code, see Documenting Python Code: A Complete Guide by James Mertz.

Whitespace in Expressions and Statements

“Sparse is better than dense.”

— The Zen of Python

Whitespace can be very helpful in expressions and statements when used properly. If there is not enough whitespace, then code can be difficult to read, as it’s all bunched together. If there’s too much whitespace, then it can be difficult to visually combine related terms in a statement.

Whitespace Around Binary Operators

Surround the following binary operators with a single space on either side:

Assignment operators (=, +=, -=, and so forth)
Comparisons (==, !=, >, <. >=, <=) and (is, is not, in, not in)
Booleans (and, not, or)

Note: When = is used to assign a default value to a function argument, do not surround it with spaces.

# Recommended
def function(default_parameter=5):
    # ...


# Not recommended
def function(default_parameter = 5):
    # ...

When there’s more than one operator in a statement, adding a single space before and after each operator can look confusing. Instead, it is better to only add whitespace around the operators with the lowest priority, especially when performing mathematical manipulation. Here are a couple examples:

# Recommended
y = x**2 + 5
z = (x+y) * (x-y)

# Not Recommended
y = x ** 2 + 5
z = (x + y) * (x - y)

You can also apply this to if statements where there are multiple conditions:

# Not recommended
if x > 5 and x % 2 == 0:
    print('x is larger than 5 and divisible by 2!')

In the above example, the and operator has lowest priority. It may therefore be clearer to express the if statement as below:

# Recommended
if x>5 and x%2==0:
    print('x is larger than 5 and divisible by 2!')

You are free to choose which is clearer, with the caveat that you must use the same amount of whitespace either side of the operator.

The following is not acceptable:

# Definitely do not do this!
if x >5 and x% 2== 0:
    print('x is larger than 5 and divisible by 2!')

In slices, colons act as a binary operators. Therefore, the rules outlined in the previous section apply, and there should be the same amount of whitespace either side. The following examples of list slices are valid:

list[3:4]

# Treat the colon as the operator with lowest priority
list[x+1 : x+2]

# In an extended slice, both colons must be
# surrounded by the same amount of whitespace
list[3:4:5]
list[x+1 : x+2 : x+3]

# The space is omitted if a slice parameter is omitted
list[x+1 : x+2 :]

In summary, you should surround most operators with whitespace. However, there are some caveats to this rule, such as in function arguments or when you’re combining multiple operators in one statement.

When to Avoid Adding Whitespace

In some cases, adding whitespace can make code harder to read. Too much whitespace can make code overly sparse and difficult to follow. PEP 8 outlines very clear examples where whitespace is inappropriate.

The most important place to avoid adding whitespace is at the end of a line. This is known as trailing whitespace. It is invisible and can produce errors that are difficult to trace.

The following list outlines some cases where you should avoid adding whitespace:

Immediately inside parentheses, brackets, or braces:

# Recommended
my_list = [1, 2, 3]

# Not recommended
my_list = [ 1, 2, 3, ]

Before a comma, semicolon, or colon:

x = 5
y = 6

# Recommended
print(x, y)

# Not recommended
print(x , y)

Before the open parenthesis that starts the argument list of a function call:

def double(x):
    return x * 2

# Recommended
double(3)

# Not recommended
double (3)

Before the open bracket that starts an index or slice:

# Recommended
list[3]

# Not recommended
list [3]

Between a trailing comma and a closing parenthesis:

# Recommended
tuple = (1,)

# Not recommended
tuple = (1, )

To align assignment operators:

# Recommended
var1 = 5
var2 = 6
some_long_var = 7

# Not recommended
var1          = 5
var2          = 6
some_long_var = 7

Make sure that there is no trailing whitespace anywhere in your code. There are other cases where PEP 8 discourages adding extra whitespace, such as immediately inside brackets, as well as before commas and colons. You should also never add extra whitespace in order to align operators.

Programming Recommendations

“Simple is better than complex.”

— The Zen of Python

You will often find that there are several ways to perform a similar action in Python (and any other programming language for that matter). In this section, you’ll see some of the suggestions PEP 8 provides to remove that ambiguity and preserve consistency.

Don’t compare boolean values to True or False using the equivalence operator. You’ll often need to check if a boolean value is True or False. When doing so, it is intuitive to do this with a statement like the one below:

# Not recommended
my_bool = 6 > 5
if my_bool == True:
    return '6 is bigger than 5'

The use of the equivalence operator, ==, is unnecessary here. bool can only take values True or False. It is enough to write the following:

# Recommended
if my_bool:
    return '6 is bigger than 5'

This way of performing an if statement with a boolean requires less code and is simpler, so PEP 8 encourages it.

Use the fact that empty sequences are falsy in if statements. If you want to check whether a list is empty, you might be tempted to check the length of the list. If the list is empty, it’s length is 0 which is equivalent to False when used in an if statement. Here’s an example:

# Not recommended
my_list = []
if not len(my_list):
    print('List is empty!')

However, in Python any empty list, string, or tuple is falsy. We can therefore come up with a simpler alternative to the above:

# Recommended
my_list = []
if not my_list:
    print('List is empty!')

While both examples will print out List is empty!, the second option is simpler, so PEP 8 encourages it.

Use is not rather than not ... is in if statements. If you are trying to check whether a variable has a defined value, there are two options. The first is to evaluate an if statement with x is not None, as in the example below:

# Recommended
if x is not None:
    return 'x exists!'

A second option would be to evaluate x is None and then have an if statement based on not the outcome:

# Not recommended
if not x is None:
    return 'x exists!'

While both options will be evaluated correctly, the first is simpler, so PEP 8 encourages it.

Don’t use if x: when you mean if x is not None:. Sometimes, you may have a function with arguments that are None by default. A common mistake when checking if such an argument, arg, has been given a different value is to use the following:

# Not Recommended
if arg:
    # Do something with arg...

This code checks that arg is truthy. Instead, you want to check that arg is not None, so it would be better to use the following:

# Recommended
if arg is not None:
    # Do something with arg...

The mistake being made here is assuming that not None and truthy are equivalent. You could have set arg = []. As we saw above, empty lists are evaluated as falsy in Python. So, even though the argument arg has been assigned, the condition is not met, and so the code in the body of the if statement will not be executed.

Use .startswith() and .endswith() instead of slicing. If you were trying to check if a string word was prefixed, or suffixed, with the word cat, it might seem sensible to use list slicing. However, list slicing is prone to error, and you have to hardcode the number of characters in the prefix or suffix. It is also not clear to someone less familiar with Python list slicing what you are trying to achieve:

# Not recommended
if word[:3] == 'cat':
    print('The word starts with "cat"')

However, this is not as readable as using .startswith():

# Recommended
if word.startswith('cat'):
    print('The word starts with "cat"')

Similarly, the same principle applies when you’re checking for suffixes. The example below outlines how you might check whether a string ends in jpg:

# Not recommended
if file_name[-3:] == 'jpg':
    print('The file is a JPEG')

While the outcome is correct, the notation is a bit clunky and hard to read. Instead, you could use .endswith() as in the example below:

# Recommended
if file_name.endswith('jpg'):
    print('The file is a JPEG')

As with most of these programming recommendations, the goal is readability and simplicity. In Python, there are many different ways to perform the same action, so guidelines on which methods to chose are helpful.

When to Ignore PEP 8

The short answer to this question is never. If you follow PEP 8 to the letter, you can guarantee that you’ll have clean, professional, and readable code. This will benefit you as well as collaborators and potential employers.

However, some guidelines in PEP 8 are inconvenient in the following instances:

If complying with PEP 8 would break compatibility with existing software
If code surrounding what you’re working on is inconsistent with PEP 8
If code needs to remain compatible with older versions of Python

Tips and Tricks to Help Ensure Your Code Follows PEP 8

There is a lot to remember to make sure your code is PEP 8 compliant. It can be a tall order to remember all these rules when you’re developing code. It’s particularly time consuming to update past projects to be PEP 8 compliant. Luckily, there are tools that can help speed up this process. There are two classes of tools that you can use to enforce PEP 8 compliance: linters and autoformatters.

Linters

Linters are programs that analyze code and flag errors. They provide suggestions on how to fix the error. Linters are particularly useful when installed as extensions to your text editor, as they flag errors and stylistic problems while you write. In this section, you’ll see an outline of how the linters work, with links to the text editor extensions at the end.

The best linters for Python code are the following:

pycodestyle is a tool to check your Python code against some of the style conventions in PEP 8.

Install pycodestyle using pip:

$ pip install pycodestyle

You can run pycodestyle from the terminal using the following command:

$ pycodestyle code.py
code.py:1:17: E231 missing whitespace after ','
code.py:2:21: E231 missing whitespace after ','
code.py:6:19: E711 comparison to None should be 'if cond is None:'

flake8 is a tool that combines a debugger, pyflakes, with pycodestyle.

Install flake8 using pip:

$ pip install flake8

Run flake8 from the terminal using the following command:

$ flake8 code.py
code.py:1:17: E231 missing whitespace after ','
code.py:2:21: E231 missing whitespace after ','
code.py:3:17: E999 SyntaxError: invalid syntax
code.py:6:19: E711 comparison to None should be 'if cond is None:'

An example of the output is also shown.

Note: The extra line of output indicates a syntax error.

These are also available as extensions for Atom, Sublime Text, Visual Studio Code, and VIM. You can also find guides on setting up Sublime Text and VIM for Python development, as well as an overview of some popular text editors at Real Python.

Autoformatters

Autoformatters are programs that refactor your code to conform with PEP 8 automatically. Once such program is black, which autoformats code following most of the rules in PEP 8. One big difference is that it limits line length to 88 characters, rather than 79. However, you can overwrite this by adding a command line flag, as you’ll see in an example below.

Install black using pip. It requires Python 3.6+ to run:

$ pip install black

It can be run via the command line, as with the linters. Let’s say you start with the following code that isn’t PEP 8 compliant in a file called code.py:

for i in range(0,3):
    for j in range(0,3):
        if (i==2):
            print(i,j)

You can then run the following command via the command line:

$ black code.py
reformatted code.py
All done! ✨ 🍰 ✨

code.py will be automatically reformatted to look like this:

for i in range(0, 3):
    for j in range(0, 3):
        if i == 2:
            print(i, j)

If you want to alter the line length limit, then you can use the --line-length flag:

$ black --line-length=79 code.py
reformatted code.py
All done! ✨ 🍰 ✨

Two other autoformatters, autopep8 and yapf, perform actions that are similar to what black does.

Another Real Python tutorial, Python Code Quality: Tools & Best Practices by Alexander van Tol, gives a thorough explanation of how to use these tools.

Conclusion

You now know how to write high-quality, readable Python code by using the guidelines laid out in PEP 8. While the guidelines can seem pedantic, following them can really improve your code, especially when it comes to sharing your code with potential employers or collaborators.

In this tutorial, you learned:

What PEP 8 is and why it exists
Why you should aim to write PEP 8 compliant code
How to write code that is PEP 8 compliant

On top of all this, you also saw how to use linters and autoformatters to check your code against PEP 8 guidelines.

If you want to learn more about PEP 8, then you can read the full documentation, or visit pep8.org, which contains the same information but has been nicely formatted. In these documents, you will find the rest of the PEP 8 guidelines not covered in this tutorial.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

A Pythonista's Holiday Wish List

2018-12-17T14:00:00+00:00

It’s that time of year again when everyone is looking to get last minute gifts. Whether you’re a friend of a Python developer or one yourself, I’ve got the perfect wish list if you’re looking to get a little something special. I’ve combed through the interwebs and polled my fellow Real Python authors to find the best presents for Pythonistas.

I’ve broken up these suggestions into five categories:

Good causes
Learning resources
Python swag
Hardware
Gadgets

Donating to a Good Cause

The greatest gift that anyone can receive is the feeling of giving to someone else. Whether you want to make a donation yourself or make one in someone’s name, here are some great Python related organizations that could use a donation:

Python Software Foundation (PSF): the official organization that maintains Python and hosts the Python Conference each year
Django Software Foundation (DSF): the official organization that maintains Django, our favorite web app framework
PyLadies: an organization dedicated to helping more women become participants and leaders in the Python open-source community

Donating your money isn’t the only thing that you can do to help the Python Community. You can also donate your time as a volunteer.

Python Learning Resources

I think that one of the greatest gifts that can be given is knowledge. Want to get a good read that’s Python related? Check out The Best Python Books for a great collection of Python related books. Maybe you want something a little more interactive? Check out the courses at Real Python!

One of the best ways to learn about Python is to learn from the masters themselves. A ticket to PyCon 2019 would be the greatest learning gift of all: three days of classes, demonstrations, coding challenges, and rubbing shoulders with the people who dream in Python code! This would be the ultimate gift that would keep on giving. You’ll also get swag bags that will last you all year round.

Python Stickers, T-Shirts, and Coffee Mugs… Oh My!

We’re Pythonistas, so why not show off the programming language we love most! I love showing off my love for Python on my laptop.

Nerdlettering has got your back for all your sticker needs, and if stickers aren’t your thing, then perhaps a sweatshirt or t-shirt might be better. Also don’t forget about that old coffee mug of yours. It could probably use an upgrade as well.

Want something that sports the Real Python logo? They’ve got you covered as well.

Hardware

With the way that tech is advancing every day, every programmer needs an upgrade of their hardware. This may be because you’re running out of hard drive space, have worn out keyboards, or just want something fun. (Or maybe you want something more advanced. Everyone loves a good challenge right?) This is my recommended hardware to check out this season.

External HDD

Whether you have a massive code base or are running several hundred Docker images, you can quickly run out of space to do all your work. Good thing the cost of hard drive capacity is almost constantly dropping! Plus, with USB 3.0, it’s almost as fast or even faster than an internal hard drive. You can get 2 TB of portable space for less than $60! Need something bigger? For a little more than double the price, you can get 8 TB! I recommend sticking to the name brands like Seagate and Western Digital if you can.

A New Mechanical Keyboard

As developers, we’re using the keyboard 99% of the time we’re on the computer. That means that it’s important to have the right keyboard that feels right for you. Don’t know what to get? The folks over at the MechanicalKeyboards subreddit have created a great guide to all things mechanical keyboards.

Recommending a keyboard is like recommending underwear: you never know how it’s going to feel for anyone else, so I highly recommend that you look at the guide above. But if you want my opinion, the keyboard that I use and think feels the best is the Microsoft Sculpt Ergo Keyboard.

Headphones

Let’s face it: there’s nothing like getting into the zone with your favorite music set and crunching away at your codebase. So why not make sure that you get a good set of headphones? I love my Beyerdynamic DT 770 Pro headphones and can’t imagine coding without them. When you’re buying headphones, I recommend using this useful guide created by the helpful folks over at the headphones subreddit.

Gadgets

It’s always fun to have new toys to play with, and us Python developers have a lot to choose from. Here, we’ve collected our favorite gadgets that use Python.

Raspberry Pi

Let’s start off with a little pie—Raspberry Pi that is. There are so many things that you can do with a Raspberry Pi, including the following:

With its starting price of only $35, it’s amazing that you’re getting a multi-core, Bluetooth and WiFi enabled device! I highly recommend getting a kit to get started.

The Open Source Rover

Speaking of Raspberry Pi, how about creating your own Mars Science Lab! NASA’s Jet Propulsion Lab (JPL), the creators of the Mars Rovers, has created an Open Source replica of the Mars Science Laboratory rover that uses the Raspberry Pi.

This comes with a hefty price tag (~$2,500) and isn’t for the faint of heart as it requires purchasing each part separately and assembling the entire rover. Don’t worry about programming the rover though. That was already done by JPL in Python! If the price and difficulty don’t scare you off, then this is a great gift for anyone wanting to learn about robotics.

Vector and Cozmo Robots From Anki

If making your own robot seems a little daunting, then check out Vector and Cozmo. Vector is “the good robot” who can help out with multiple things such as a kitchen timer, taking a selfie, or telling you what the weather is going to be like. If that’s not enough, there is the Python Software Development Kit (SDK) that enables you to make Vector fully customizable!

If you’re new to programming or want to teach someone about programming robots, then Cozmo’s got you covered. With both a full Python SDK and a more simple option available, this robot is a good choice for the beginner and the seasoned coder.

DJI Drones

A little tired of being stuck on the ground? How about flying? DJI has multiple versions of their drones that you can take flight with either manually or automated using their software. If they don’t have the feature you’re looking for, how about checking out the SDK and doing it yourself! DJI has a lot of drone options available, but I’ve got my eye on the Tello, since it’s on the cheaper end and has an easier-to-use Python Library.

Happy Pythonic Holidays

Whatever you decide to get you or your Python developer friends, be sure to enjoy this holiday season and keep on Pythoning! I’m about to push checkout on my Amazon cart now and tell my wife that #PythonMadeMeDoIt.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Thonny: The Beginner-Friendly Python Editor

2018-12-12T14:00:00+00:00

Are you a Python beginner looking for a tool that can support your learning? This article is for you! Every programmer needs a place to write their code. This article will discuss an awesome tool called Thonny that will enable you to start working with Python in a beginner-friendly environment.

In this article, you’ll learn:

How to install Thonny on your computer
How to navigate Thonny’s user interface to use its built-in features
How to use Thonny to write and run your code
How to use Thonny to debug your code

By the end of this article, you’ll be comfortable with the development workflow in Thonny and ready to use it for your Python learning.

So what is Thonny? Great question!

Thonny is a free Python Integrated Development Environment (IDE) that was especially designed with the beginner Pythonista in mind. Specifically, it has a built-in debugger that can help when you run into nasty bugs, and it offers the ability to do step through expression evaluation, among other really awesome features.

Free Sample Chapter: Download a free sample chapter from the Real Python course and gain practical Python programming skills.

Installing Thonny

This article assumes that you have Python 3 installed on your computer. If not, please review Python 3 Installation & Setup.

Web Download

The web download can be accessed via a web browser by visiting the Thonny website. Once on the page, you will see a light gray box in the top right corner like this:

Once you’ve found the gray box, click the appropriate link for your operating system. This tutorial assumes you’ve downloaded version 3.0.1.

Command Line Download

You can also install Thonny via your system’s command line. On Windows, you can do this by starting a program called Command Prompt, while on macOS and Linux you start a program called Terminal. Once you’ve done that, enter the following command:

$ pip install thonny

The User Interface

Let’s make sure you understand what Thonny has to offer. Think of Thonny as the workroom in which you will create amazing Python projects. Your workroom contains a toolbox containing many tools that will enable you to be a rock star Pythonista. In this section, you’ll learn about each of the features of the UI that’ll help you use each of the tools in your Thonny toolbox.

The Code Editor and Shell

Now that you have Thonny installed, open the application. You should see a window with several icons across the top, and two white areas:

Notice the two main sections of the window. The top section is your code editor, where you will write all of your code. The bottom half is your Shell, where you will see outputs from your code.

The Icons

Across the top you’ll see several icons. Let’s explore what each of them does. You’ll see an image of the icons below, with a letter above each one. We will use these letters to talk about each of the icons:

Working our way from left to right, below is a description of each of the icons in the image.

A: The paper icon allows you to create a new file. Typically in Python you want to separate your programs into separate files. You’ll use this button later in the tutorial to create your first program in Thonny!

B: The open folder icon allows you to open a file that already exists on your computer. This might be useful if you come back to a program that you worked on previously.

C: The floppy disk icon allows you to save your code. Press this early and often. You’ll use this later to save your first Thonny Python program.

D: The play icon allows you to run your code. Remember that the code you write is meant to be executed. Running your code means you’re telling Python, “Do what I told you to do!” (In other words, “Read through my code and execute what I wrote.”)

E: The bug icon allows you to debug your code. It’s inevitable that you will encounter bugs when you’re writing code. A bug is another word for a problem. Bugs can come in many forms, sometimes appearing when you use inappropriate syntax and sometimes when your logic is incorrect.

Thonny’s bug button is typically used to spot and investigate bugs. You’ll work with this later in the tutorial. By the way, if you’re wondering why they’re called bugs, there’s also a fun story of how it came about!

F-H: The arrow icons allow you to run your programs step by step. This can be very useful when you’re debugging or, in other words, trying to find those nasty bugs in your code. These icons are used after you press the bug icon. You’ll notice as you hit each arrow, a yellow highlighted bar will indicate which line or section Python is currently evaluating:

The F arrow tells Python to take a big step, meaning jumping to the next line or block of code.
The G arrow tells Python to take a small step, meaning diving deep into each component of an expression.
The H arrow tells Python to exit out of the debugger.

I: The resume icon allows you to return to play mode from debug mode. This is useful in the instance when you no longer want to go step by step through the code, and instead want your program to finish running.

J: The stop icon allows you to stop running your code. This can be particularly useful if, let’s say, your code runs a program that opens a new window, and you want to stop that program. You’ll use the stop icon later in the tutorial.

Let’s Try It!

Get ready to write your first official Python program in Thonny:

Enter the following code into the code editor:
```
print("Hello World")
```
Click the play button to run your program.
See the output in the Shell window.
Click the play button again to see that it says hello one more time.

Congratulations! You’ve now completed your first program in Thonny! You should see Hello world! printed inside the Shell, also known as the console. This is because your program told Python to print this phrase, and the console is where you see the output of this execution.

Other UI Features

To see more of the other features that Thonny has to offer, navigate to the menu bar and select the View dropdown. You should see that Shell has a check mark next to it, which is why you see the Shell section in Thonny’s application window:

Let’s explore some of the other offerings, specifically those that will be useful to a beginning Pythonista:

Help: You’ll select the Help view if you want more information about working with Thonny. Currently this section offers more reading on the following topics: Running Programs Step-wise, how to install 3rd Party Packages, or using Scientific Python Packages.
Variables: This feature can be very valuable. A variable in Python is a value that you define in code. Variables can be numbers, strings, or other complex data structures. This section allows you to see the values assigned to all of the variables in your program.
Assistant: The Assistant is there to give you helpful hints when you hit Exceptions or other types of errors.

The other features will become useful as you advance your skills. Check them out once you get more comfortable with Thonny!

The Code Editor

Now that you have an understanding of the UI, let’s use Thonny to write another little program. In this section, you’ll go through the features of Thonny that will help guide you through your development workflow.

Write Some Code

In the code editor (top portion of the UI), add the following function:

def factorial(num):
    if num == 1:
        return 1
    else:
        return num * factorial(num - 1)

print(factorial(3))

Save Your Code

Before we move on, let’s save your program. Last time, you were prompted to do this after pressing the play button. You can also do this by clicking the blue floppy disk icon or by going to the menu bar and selecting File > Save. Let’s call the program factorial.py.

Run Your Code

In order to run your code, find and press the play icon. The output should look like this:

Debug Your Code

To truly understand what this function is doing, try the step feature. Take a few large and small steps through the function to see what is happening. Remember you can do this by pressing the arrow icons:

As you can see, the steps will show how the computer is evaluating each part of the code. Each pop up window is like a piece of scratch paper that the computer is using to compute each portion of the code. Without this awesome feature, this may have been hard to conceptualize—but now you’ve got it!

Stop Running Your Code

So far, there hasn’t been a need to hit the stop icon for this program, particularly because it exits as soon as it has executed print(). Try increasing the number being passed to the factorial function to 100:

def factorial(num):
    if num == 1:
        return 1
    else:
        return num * factorial(num - 1)

print(factorial(100))

Then step through the function. After a while, you will notice that you will be clicking for a long time to reach the end. This is a good time to use the stop button. The stop button can be really useful to stop a program that is either intentionally or unintentionally running continuously.

Find Syntax Errors in Your Code

Now that you have a simple program that works, let’s break it! By intentionally creating an error in your factorial program, you’ll be able to see how Thonny handles these types of issues.

We will be creating what is called a syntax error. A syntax error is an error that indicates that your code is syntactically incorrect. In other words, your code does not follow the proper way to write Python. When Python notices the error, it will display a syntax error to complain about your invalid code.

Above the print statement, let’s add another print statement that says print("The factorial of 100 is:"). Now let’s go ahead and create syntax errors. In the first print statement, remove the second quotation mark, and in the other remove the second parenthesis.

As you do this, you should see that Thonny will highlight your SyntaxErrors. Missing quotations are highlighted in green, and missing parenthesis are in gray:

For beginners, this is a great resource that will allow you to help spot any typos while you’re writing. Some of the most common and frustrating errors when you start programming are missing quotes and mismatched parentheses.

If you have your Assistant View turned on, you will also notice that it will give you a helpful message to guide you in the right direction when you are debugging:

As you get more comfortable with Thonny, the Assistant can be a useful tool to help you get unstuck!

The Package Manager

As you continue to learn Python, it can be quite useful to download a Python package to use inside of your code. This allows you to use code that someone else has written inside of your program.

Consider an example where you want to do some calculations in your code. Instead of writing your own calculator, you might want to use a third-party package called simplecalculator. In order to do this, you’ll use Thonny’s package manager.

The package manager will allow you to install packages that you will need to use with your program. Specifically, it allows you to add more tools to your toolbox. Thonny has the built-in benefit of handling any conflicts with other Python interpreters.

To access the package manager, go to the menu bar and select Tools > Manage Packages… This should pop open a new window with a search field. Type simplecalculator into that field and click the Search button.

The output should look similar to this:

Go ahead and click Install to install this package. You will see a small window pop up showing the system’s logs while it installs the package. Once it completes, you are ready to use simplecalculator in your code!

In the next section, you will use the simplecalculator package along with some of the other skills you’ve learned in this tutorial to create a simple calculator program.

Check Your Understanding

You’ve learned so much about Thonny so far! Here’s what you’ve learned:

Where to write your code
How to save your code
How to run your code
How to stop your code from running
Where to see your code execute
How to spot SyntaxErrors
How to install third party packages

Let’s check your understanding of these concepts.

Now that you have simplecalculator installed, let’s create a simple program that will use this package. You’ll also use this as an opportunity to check that you understand some of the UI and development features that you’ve learned thus far in the tutorial.

Part 1: Create a File, Add Some Code, and Understand the Code

In Part 1, you will create a file, and add some code to it! Do your best to try to dig into what the code is actually doing. If you get stuck, check out the Take a Deeper Look window. Let’s get started:

Start a new file.
Add the following code into your Thonny code editor:

 1 from calculator.simple import SimpleCalculator
 2 
 3 my_calculator = SimpleCalculator()  
 4 my_calculator.run('2 * 2')
 5 print(my_calculator.lcd)

This code will print out the result of 2 * 2 to the Thonny Shell in the main UI. To understand what each part of the code is doing, check out the Take a Deeper Look section below.

Line 1: This code imports the library calculator inside of the package called simplecalculator. From this library, we import the class called SimpleCalculator from a file called simple.py. You can see the code here.
Lines 2: This is a blank line behind code blocks, which is generally a preferred style. Read more about Python Code Quality in this article.
Line 3: Here we create an instance of the class SimpleCalculator and assign it to a variable called my_calculator. This can be used to run different calculators. If you’re new to classes, you can learn more about object-oriented programming here.
Line 4: Here we have the calculator run the operation 2 * 2 by calling run() and passing in the expression as a string.
Line 5: Here we print the result of the calculation. You’ll notice in order to get the most recent calculation result, we must access the attribute called lcd.

Great! Now that you know exactly what your calculator code is doing, let’s move on to running this code!

Part 2: Save the File, View the Variables, and Run Your Code

Now it’s time to save and run your code. In this section, you’ll make use of two of the icons we reviewed earlier:

Save your new file as calculations.py.
Open the Variables window and make note of the two variables listed. You should see SimpleCalculator and my_calculator. This section also gives you insight into the value that each variable is pointing to.
Run your code! You should see 4.0 in the output:

Great job! Next you’ll explore how Thonny’s debugger can help you to better understand this code.

Other Great Beginner Features

As you get more comfortable with Thonny, the features in this section will come in quite handy.

Debugging

Using your calculations.py script, you’re going to use the debugger to investigate what is happening. Update your code in calculations.py to the following:

from calculator.simple import SimpleCalculator

def create_add_string(x, y):
    '''Returns a string containing an addition expression.'''
    return 'x + y'

my_calculator = SimpleCalculator()  
my_calculator.run(create_add_string(2, 2))
print(my_calculator.lcd)

Hit the save icon to save this version.

You’ll notice the code has a new function called create_add_string(). If you’re unfamiliar with Python functions, learn more in this awesome Real Python course!

As you inspect the function, you may notice why this script will not work as expected. If not, that’s okay! Thonny is going to help you see exactly what is going on, and squash that bug! Go ahead and run your program and see what happens. The Shell output should be the following:

>>> %Run calculations.py
0

Oh no! Now you can see there is a bug in your program. The answer should be 4! Next, you’ll use Thonny’s debugger to find the bug.

Let’s Try It!

Now that we have a bug in our program, this is a great chance to use Thonny’s debugging features:

Click the bug icon at the top of the window. This enters debugger mode.
You should see the import statements highlighted. Click the small step arrow icon, the yellow arrow in the middle. Keep pressing this to see how the debugger works. You should notice that it highlights each step that Python takes to evaluate your program. Once it hits create_add_string(), you should see a new window pop up.
Examine the pop up window carefully. You should see that it shows the values for x and y. Keep pressing the small step icon until you see the value that Python will return to your program. It will be enclosed in a light-blue box: Oh no! There’s the bug! It looks like Python will return a string containing the letters x and y (meaning 'x + y' and not a string containing the values of those variables, like '2 + 2', which is what the calculator is expecting.) Each time you see a light-blue box, you can think of this as Python replacing subexpressions with their values, step by step. The pop up window can be thought of as a piece of scratch paper that Python uses to figure out those values. Continue to step through the program to see how this bug results in a calculation of 0.
The bug here has to do with string formatting. If you are unfamiliar with string formatting, check out this article on Python String Formatting Best Practices. Inside create_add_string(), the f-string formatting method should be used. Update this function to the following:
```
def create_add_string(x, y):
    '''Returns a string containing an addition expression.'''
    return f'{x} + {y}'
```
Run your program again. You should see the following output:
```
>>> %Run calculations.py
4.0
```

Success! You have just demonstrated how the step-by-step debugger can help you find a problem in your code! Next you’ll learn about some other fun Thonny features.

Variable Scope Highlighting

Thonny offers variable highlighting to remind you that the same name doesn’t always mean the same variable. In order for this feature to work, on the menu bar, go to Thonny > Preferences and ensure that Highlight matching names is checked.

Notice in the code snippet below, that create_add_string() now has a new variable called my_calculator, though this is not the same as the my_calculator on lines 10 and 11. You should be able to tell because Thonny highlights the variables that reference the same thing. This my_calculator inside the function only exists within the scope of that function, which is why it is not highlighted when the cursor is on the other my_calculator variable on line 10:

This feature can really help you avoid typos and understand the scope of your variables.

Code Completion

Thonny also offers code completion for APIs. Notice in the snapshot below how pressing the Tab key shows the methods available from the random library:

This can be very useful when you’re working with libraries and don’t want to look at the documentation to find a method or attribute name.

Working on a Pre-Existing Project

Now that you’ve learned the basic features of Thonny, let’s explore how you can use it to work on a pre-existing project.

Find a File on Your Computer

Opening a file on your computer is as easy as going to the menu bar, selecting File > Open, and using your browser to navigate to the file. You can also use the open folder icon at the top of the screen to do this as well.

If you have a requirements.txt file and pip locally installed, you can pip install these from the Thonny system Shell. If you don’t have pip installed, remember you can use the Package Manager to install it:

$ pip install -r requirements.txt

Work on a Project From Github

Now that you are a Thonny expert, you can use it to work on the exercises from Real Python Course 1: Introduction to Python:

Navigate to the Real Python GitHub repo called book1-exercises.
Click the green button labeled Clone or download and select Download Zip.
Click the opening folder icon to navigate and find the downloaded files. You should find a folder called book1-exercises1.
Open one of the files and start working!

This is useful because there are tons of cool projects available on GitHub!

Conclusion

Awesome job getting through this tutorial on Thonny!

You can now start using Thonny to write, debug, and run Python code! If you like Thonny, you might also like some of the other IDEs we’ve listed in Python IDEs and Code Editors (Guide).

Thonny is actively maintained, and new features are being added all the time. There are several awesome new features that are currently in beta that can be found on the Thonny Blog. Thonny’s main development takes place at the Institute of Computer Science of the University of Tartu, Estonia, as well as by contributors around the world.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python Community Interview With Brian Peterson

2018-12-10T14:00:00+00:00

To date, I’ve interviewed people you’ve likely heard of before from the Python community. But this column isn’t just about interviewing the rock stars and core devs. It’s also a means to shine light on the huge contributions to the community that can often go unthanked and overlooked. As such, I present to you Brian Peterson.

Brian is a project manager by day, and by night he’s one of the moderators of the Pythonista Café, a peer-to-peer learning community for Pythonistas. In our interview, we talk about how Python helps him in his role as a project manager, and how moderating a large forum for Python enthusiasts has impacted his coding chops. Let’s dig in!

Ricky: Welcome to Real Python! Let’s get started with the question I ask everyone. How did you get into programming, and when did you start using Python?

Brian: Well… thinking back, my first real exposure to programming was testing near object detection system prototypes using an old HP-87. The program was used to move and change target and rotating antenna positions along test tracks inside an automotive laboratory anechoic chamber, all while collecting and processing data from a spectrum analyzer. Seeing the code translate into motion did it for me—something real, and tangible. I was hooked on using programming as an R&D tool to make things come alive.

Over the years, I started spending more time with Linux automation, control systems, data collection and analysis, which naturally led to spending more time writing C code, and then it was like, “Hey, Python is already on the system. Why not take it for a spin?”

I think it was Python 2.3 at that time. I liked Python because it felt natural, and I didn’t have to throw away all my C knowledge. It had some functional programming sprinkled in, and an impressive set of engineering and scientific libraries. To top it off, I could use the same language seamlessly across scripting, interactive data analysis, and writing code. And Python syntax, well, it can be so darned readable that personally, I think it’s hard not to like it. I was hooked.

Ricky: Working in project management and using Python might not seem a natural fit for some people. How do you use Python to help you in the project management office, and what tools and libraries are the most helpful for you on a day-to-day basis?

Brian: Yeah, at first glance Python may not seem like a natural fit. This is in large part due to the way project management is commonly taught—like cooking from a recipe book rather than learning and understanding the culinary arts.

And cookie-cutter project management tools are like the microwave ovens. Yeah, sure, use ‘em for the prepacked, boxed, canned stuff, but for every meal? Seriously? Python also starts making sense when you’re considering different flavors of project management:

Traditional project managers use template-driven approaches.
Agile practitioners use story-driven approaches.
Adaptive practitioners blend existing, new, emerging approaches and creativity to develop the best fit for each unique project or group of projects.

So while off-the-shelf project management tools and services are an excellent fit for traditional projects, and a good fit for some Agile projects, adaptive project management calls for a different breed. In other words, rather than force all projects to fit a tool, why not use Python to construct a toolbox filled with best-of-breed tools to solve different types of problems for different kinds of projects?

It’s also worth considering that modern day Project Management Offices (PMO) don’t just manage projects: there’s ideation, portfolio management, road mapping, strategic planning, KPI monitoring, communications, governance, dashboards, stakeholder portals, working closely with DevOps, business analysts, architects, developers, UX designers, CX folks, subject matter experts, other technologists, and the list goes on and on.

Now imagine a vendor who knows nothing about your particular projects or PMO services with a big box of Legos gluing pieces into place and selling it to you. Yeah, it’s probably a nice contraption, or it wouldn’t be on the market for long.

But compare it to owning your own big box of Legos (Python and goodies), where you can add Legos to pre-built contraptions, use Legos to bridge multiple contraptions together, use them with existing Legos already lying around the organization, or build your own and mix and match any or all of the above.

And to answer the second part of the question, I currently use Python and RESTful APIs to interact with a variety of flexible SaaS tools, for example, ProductPlan, Smartsheet (with the dashboard extension), Airtable, and a commercial ticketing system, and enterprise visualization software. Python makes it easy to wire these tools together in creative ways and fill gaps in functionality to solve many different kinds of problems.

Ricky: How have those tools changed the way you work? And how do you see them changing going forward? Will Python continue to play a role in the future of project management offices?

Brian: I think the most significant change is a sense of freedom, the ability to creatively solve problems, focus on what’s essential, and leverage modern solutions rather than being chained to tools, falling into tool ruts, or suffering from tool rot. As an added bonus, Python makes project management fun and refreshing.

I’m not sure that Python will become mainstream in project management, but I do think it can give project managers who learn it a unique advantage in the automation of tedious tasks, solving complex problems, adapting to the evolution of organizations, and providing valuable outside-the-box PMO services.

Ricky: Do you have any Python or tech related side projects you’re working on right now?

Brian: Yes, Road Maps to the Future is a project in partnership with a network of volunteers and in cooperation with agencies to help pilot it. With over 80,000 local governments in the United States alone, I believe there needs to be a better way to map out where we are, where we are going in the future, and the best ways to get there.

In Python, we have an amazing assortment of tools for doing fantastic things with data, and we’re getting better at collecting discrete data.

But there are opportunities in the exploration and non-linear collection of ideas, the identification of unseen problems, gaps in services, areas for improvement in the overall customer experience, the development of creative and viable alternative solutions to these problems, and then translating them into successful projects with demonstrable benefits.

The road maps to the future are not just compilations of random ideas with new ways to collect information, but cohesive collections of core elements, processes, and partnerships necessary to propel the ideas into action through the organization of projects.

To make it sustainable, it has to be discoverable, simple, quick and fun, so a lot of extra focus is being put into the UX side of the project. The current design uses Python, Vue.js, Azure Cosmos DB, Service Bus and Cognitive Services. It’s still in early development. Project information, details, and code will be released into the public domain next year.

Ricky: One of the reasons I was keen to interview you is because of your contribution to the Python community. You are one of two moderators in the Pythonista Café, a peer-to-peer learning community for Python enthusiasts. How has your journey been from being a member to a moderator, and how has it influenced your Python chops?

Brian: It’s been great! Dan Bader has done such a fantastic job, along with you, Jennifer (co-moderator), the Real Python team, and especially all the amazing contributors from a wide range of backgrounds, geographical locations, and levels of experience who have made the Café into the lively forum it is today.

It really is truly community-driven from the ground up. It has a fun, productive, open-source vibe without fear of public shaming. It’s okay to make mistakes and learn. You can post stuff without having to worry about how the whole world will interpret it or misinterpret it.

I initially joined the Café for several reasons. It’s like a non-cliquey 24/7/365 Python meetup that you can drop in and out of at any time. As forum members have mentioned, it has a much better signal-to-noise ratio than the public forums, not to mention freedom from ads and social media algorithms that manipulate behaviors.

It’s very welcoming and ultra-Python friendly. The passion for programming is seriously contagious, which leads to your question on how it has influenced my Python chops. The answer is that it has made Python even more fun, and I’m now even more committed and focused to continuous learning in Python and its ecosystem. It has also increased my excitement about the possibilities of what can be done with Python in the future.

Ricky: Now for my last question. What other hobbies and interests do you have, aside from Python? Any you’d like to share and/or plug?

Brian: Outside of work and Python, I like to mix it up a lot. I currently work in the Panhandle and spend my off-time on the First Coast, so lately one of my favorite things to do is wandering around the south kayaking the many rivers and swamps; under certain conditions, it’s absolutely surreal. You can read and watch all you want about swamps, but there’s no substitute for the real thing.

Each year I also pick out one or two new subjects to research and study for fun. I am an amateur radio operator, I love music, and I’m an avid reader of books, particularly history, survival stories, and science fiction.

I’m also blessed to have an amazing wife and son. Outside of my projects, my wife keeps things lively with home projects, going to festivals and other events, helping animal rescues, along with exploring new areas and hiking with the dogs. Boredom is an alien concept—it definitely keeps life fun.

Thank you, Brian, for joining me this week. If you’re looking for a friendly Python place to hang your metaphorical hat, you can find the Pythonista Café here. Just be sure to say hi to Brian when you get there.

Do you know an unsung hero in the Python community? If they’d like me to interview them in the future, I can be reached in the comments below, or you can send me a message on Twitter.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Sending Emails With Python

2018-12-05T14:00:00+00:00

You probably found this tutorial because you want to send emails using Python. Perhaps you want to receive email reminders from your code, send a confirmation email to users when they create an account, or send emails to members of your organization to remind them to pay their dues. Sending emails manually is a time-consuming and error-prone task, but it’s easy to automate with Python.

In this tutorial you’ll learn how to:

Set up a secure connection using SMTP_SSL() and .starttls()
Use Python’s built-in smtplib library to send basic emails
Send emails with HTML content and attachments using the email package
Send multiple personalized emails using a CSV file with contact data
Use the Yagmail package to send email through your Gmail account using only a few lines of code

You’ll find a few transactional email services at the end of this tutorial, which will come in useful when you want to send a large number of emails.

Free Bonus: Click here to get access to a chapter from Python Tricks: The Book that shows you Python's best practices with simple examples you can apply instantly to write more beautiful + Pythonic code.

Getting Started

Python comes with the built-in smtplib module for sending emails using the Simple Mail Transfer Protocol (SMTP). smtplib uses the RFC 821 protocol for SMTP. The examples in this tutorial will use the Gmail SMTP server to send emails, but the same principles apply to other email services. Although the majority of email providers use the same connection ports as the ones in this tutorial, you can run a quick Google search to confirm yours.

To get started with this tutorial, set up a Gmail account for development, or set up an SMTP debugging server that discards emails you send and prints them to the command prompt instead. Both options are laid out for you below. A local SMTP debugging server can be useful for fixing any issues with email functionality and ensuring your email functions are bug-free before sending out any emails.

Option 1: Setting up a Gmail Account for Development

If you decide to use a Gmail account to send your emails, I highly recommend setting up a throwaway account for the development of your code. This is because you’ll have to adjust your Gmail account’s security settings to allow access from your Python code, and because there’s a chance you might accidentally expose your login details. Also, I found that the inbox of my testing account rapidly filled up with test emails, which is reason enough to set up a new Gmail account for development.

A nice feature of Gmail is that you can use the + sign to add any modifiers to your email address, right before the @ sign. For example, mail sent to my+person1@gmail.com and my+person2@gmail.com will both arrive at my@gmail.com. When testing email functionality, you can use this to emulate multiple addresses that all point to the same inbox.

To set up a Gmail address for testing your code, do the following:

Create a new Google account.
Turn Allow less secure apps to ON. Be aware that this makes it easier for others to gain access to your account.

If you don’t want to lower the security settings of your Gmail account, check out Google’s documentation on how to gain access credentials for your Python script, using the OAuth2 authorization framework.

Option 2: Setting up a Local SMTP Server

You can test email functionality by running a local SMTP debugging server, using the smptd module that comes pre-installed with Python. Rather than sending emails to the specified address, it discards them and prints their content to the console. Running a local debugging server means it’s not necessary to deal with encryption of messages or use credentials to log in to an email server.

You can start a local SMTP debugging server by typing the following in Command Prompt:

$ python -m smtpd -c DebuggingServer -n localhost:1025

On Linux, use the same command preceded by sudo.

Any emails sent through this server will be discarded and shown in the terminal window as a bytes object for each line:

---------- MESSAGE FOLLOWS ----------
b'X-Peer: ::1'
b''
b'From: my@address.com'
b'To: your@address.com'
b'Subject: a local test mail'
b''
b'Hello there, here is a test email'
------------ END MESSAGE ------------

For the rest of the tutorial, I’ll assume you’re using a Gmail account, but if you’re using a local debugging server, just make sure to use localhost as your SMTP server and use port 1025 rather than port 465 or 587. Besides this, you won’t need to use login() or encrypt the communication using SSL/TLS.

Sending a Plain-Text Email

Before we dive into sending emails with HTML content and attachments, you’ll learn to send plain-text emails using Python. These are emails that you could write up in a simple text editor. There’s no fancy stuff like text formatting or hyperlinks. You’ll learn that a bit later.

Starting a Secure SMTP Connection

When you send emails through Python, you should make sure that your SMTP connection is encrypted, so that your message and login credentials are not easily accessed by others. SSL (Secure Sockets Layer) and TLS (Transport Layer Security) are two protocols that can be used to encrypt an SMTP connection. It’s not necessary to use either of these when using a local debugging server.

There are two ways to start a secure connection with your email server:

Start an SMTP connection that is secured from the beginning using SMTP_SSL().
Start an unsecured SMTP connection that can then be encrypted using .starttls().

In both instances, Gmail will encrypt emails using TLS, as this is the more secure successor of SSL. As per Python’s Security considerations, it is highly recommended that you use create_default_context() from the ssl module. This will load the system’s trusted CA certificates, enable host name checking and certificate validation, and try to choose reasonably secure protocol and cipher settings.

If you want to check the encryption for an email in your Gmail inbox, go to More → Show original to see the encryption type listed under the Received header.

smtplib is Python’s built-in module for sending emails to any Internet machine with an SMTP or ESMTP listener daemon.

I’ll show you how to use SMTP_SSL() first, as it instantiates a connection that is secure from the outset and is slightly more concise than the .starttls() alternative. Keep in mind that Gmail requires that you connect to port 465 if using SMTP_SSL(), and to port 587 when using .starttls().

Option 1: Using `SMTP_SSL()`

The code example below creates a secure connection with Gmail’s SMTP server, using the SMTP_SSL() of smtplib to initiate a TLS-encrypted connection. The default context of ssl validates the host name and its certificates and optimizes the security of the connection. Make sure to fill in your own email address instead of my@gmail.com:

import smtplib, ssl

port = 465  # For SSL
password = input("Type your password and press enter: ")

# Create a secure SSL context
context = ssl.create_default_context()

with smtplib.SMTP_SSL("smtp.gmail.com", port, context=context) as server:
    server.login("my@gmail.com", password)
    # TODO: Send email here

Using with smtplib.SMTP_SSL() as server: makes sure that the connection is automatically closed at the end of the indented code block. If port is zero, or not specified, .SMTP_SSL() will use the standard port for SMTP over SSL (port 465).

It’s not safe practice to store your email password in your code, especially if you intend to share it with others. Instead, use input() to let the user type in their password when running the script, as in the example above. If you don’t want your password to show on your screen when you type it, you can import the getpass module and use .getpass() instead for blind input of your password.

Option 2: Using `.starttls()`

Instead of using .SMTP_SSL() to create a connection that is secure from the outset, we can create an unsecured SMTP connection and encrypt it using .starttls().

To do this, create an instance of smtplib.SMTP, which encapsulates an SMTP connection and allows you access to its methods. I recommend defining your SMTP server and port at the beginning of your script to configure them easily.

The code snippet below uses the construction server = SMTP(), rather than the format with SMTP() as server: which we used in the previous example. To make sure that your code doesn’t crash when something goes wrong, put your main code in a try block, and let an except block print any error messages to stdout:

import smtplib, ssl

smtp_server = "smtp.gmail.com"
port = 587  # For starttls
sender_email = "my@gmail.com"
password = input("Type your password and press enter: ")

# Create a secure SSL context
context = ssl.create_default_context()

# Try to log in to server and send email
try:
    server = smtplib.SMTP(smtp_server,port)
    server.ehlo() # Can be omitted
    server.starttls(context=context) # Secure the connection
    server.ehlo() # Can be omitted
    server.login(sender_email, password)
    # TODO: Send email here
except Exception as e:
    # Print any error messages to stdout
    print(e)
finally:
    server.quit()

To identify yourself to the server, .helo() (SMTP) or .ehlo() (ESMTP) should be called after creating an .SMTP() object, and again after .starttls(). This function is implicitly called by .starttls() and .sendmail() if needed, so unless you want to check the SMTP service extensions of the server, it is not necessary to use .helo() or .ehlo() explicitly.

Sending Your Plain-text Email

After you initiated a secure SMTP connection using either of the above methods, you can send your email using .sendmail(), which pretty much does what it says on the tin:

server.sendmail(sender_email, receiver_email, message)

I recommend defining the email addresses and message content at the top of your script, after the imports, so you can change them easily:

sender_email = "my@gmail.com"
receiver_email = "your@gmail.com"
message = """\
Subject: Hi there

This message is sent from Python."""

# Send email here

The message string starts with "Subject: Hi there" followed by two newlines (\n). This ensures Hi there shows up as the subject of the email, and the text following the newlines will be treated as the message body.

The code example below sends a plain-text email using SMTP_SSL():

import smtplib, ssl

port = 465  # For SSL
smtp_server = "smtp.gmail.com"
sender_email = "my@gmail.com"  # Enter your address
receiver_email = "your@gmail.com"  # Enter receiver address
password = input("Type your password and press enter: ")
message = """\
Subject: Hi there

This message is sent from Python."""

context = ssl.create_default_context()
with smtplib.SMTP_SSL(smtp_server, port, context=context) as server:
    server.login(sender_email, password)
    server.sendmail(sender_email, receiver_email, message)

For comparison, here is a code example that sends a plain-text email over an SMTP connection secured with .starttls(). The server.ehlo() lines may be omitted, as they are called implicitly by .starttls() and .sendmail(), if required:

import smtplib, ssl

port = 587  # For starttls
smtp_server = "smtp.gmail.com"
sender_email = "my@gmail.com"
receiver_email = "your@gmail.com"
password = input("Type your password and press enter:")
message = """\
Subject: Hi there

This message is sent from Python."""

context = ssl.create_default_context()
with smtplib.SMTP(smtp_server, port) as server:
    server.ehlo()  # Can be omitted
    server.starttls(context=context)
    server.ehlo()  # Can be omitted
    server.login(sender_email, password)
    server.sendmail(sender_email, receiver_email, message)

Sending Fancy Emails

Python’s built-in email package allows you to structure more fancy emails, which can then be transferred with smptlib as you have done already. Below, you’ll learn how use the email package to send emails with HTML content and attachments.

Including HTML Content

If you want to format the text in your email (bold, italics, and so on), or if you want to add any images, hyperlinks, or responsive content, then HTML comes in very handy. Today’s most common type of email is the MIME (Multipurpose Internet Mail Extensions) Multipart email, combining HTML and plain-text. MIME messages are handled by Python’s email.mime module. For a detailed description, check the documentation.

As not all email clients display HTML content by default, and some people choose only to receive plain-text emails for security reasons, it is important to include a plain-text alternative for HTML messages. As the email client will render the last multipart attachment first, make sure to add the HTML message after the plain-text version.

In the example below, our MIMEText() objects will contain the HTML and plain-text versions of our message, and the MIMEMultipart("alternative") instance combines these into a single message with two alternative rendering options:

import smtplib, ssl
from email.mime.text import MIMEText
from email.mime.multipart import MIMEMultipart

sender_email = "my@gmail.com"
receiver_email = "your@gmail.com"
password = input("Type your password and press enter:")

message = MIMEMultipart("alternative")
message["Subject"] = "multipart test"
message["From"] = sender_email
message["To"] = receiver_email

# Create the plain-text and HTML version of your message
text = """\
Hi,
How are you?
Real Python has many great tutorials:
www.realpython.com"""
html = """\
<html>
  <body>
    <p>Hi,<br>
       How are you?<br>
       <a href="http://www.realpython.com">Real Python</a> 
       has many great tutorials.
    </p>
  </body>
</html>
"""

# Turn these into plain/html MIMEText objects
part1 = MIMEText(text, "plain")
part2 = MIMEText(html, "html")

# Add HTML/plain-text parts to MIMEMultipart message
# The email client will try to render the last part first
message.attach(part1)
message.attach(part2)

# Create secure connection with server and send email
context = ssl.create_default_context()
with smtplib.SMTP_SSL("smtp.gmail.com", 465, context=context) as server:
    server.login(sender_email, password)
    server.sendmail(
        sender_email, receiver_email, message.as_string()
    )

In this example, you first define the plain-text and HTML message as string literals, and then store them as plain/html MIMEText objects. These can then be added in this order to the MIMEMultipart("alternative") message and sent through your secure connection with the email server. Remember to add the HTML message after the plain-text alternative, as email clients will try to render the last subpart first.

Adding Attachments Using the `email` Package

In order to send binary files to an email server that is designed to work with textual data, they need to be encoded before transport. This is most commonly done using base64, which encodes binary data into printable ASCII characters.

The code example below shows how to send an email with a PDF file as an attachment:

import email, smtplib, ssl

from email import encoders
from email.mime.base import MIMEBase
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText

subject = "An email with attachment from Python"
body = "This is an email with attachment sent from Python"
sender_email = "my@gmail.com"
receiver_email = "your@gmail.com"
password = input("Type your password and press enter:")

# Create a multipart message and set headers
message = MIMEMultipart()
message["From"] = sender_email
message["To"] = receiver_email
message["Subject"] = subject
message["Bcc"] = receiver_email  # Recommended for mass emails

# Add body to email
message.attach(MIMEText(body, "plain"))

filename = "document.pdf"  # In same directory as script

# Open PDF file in binary mode
with open(filename, "rb") as attachment:
    # Add file as application/octet-stream
    # Email client can usually download this automatically as attachment
    part = MIMEBase("application", "octet-stream")
    part.set_payload(attachment.read())

# Encode file in ASCII characters to send by email    
encoders.encode_base64(part)

# Add header as key/value pair to attachment part
part.add_header(
    "Content-Disposition",
    f"attachment; filename= {filename}",
)

# Add attachment to message and convert message to string
message.attach(part)
text = message.as_string()

# Log in to server using secure context and send email
context = ssl.create_default_context()
with smtplib.SMTP_SSL("smtp.gmail.com", 465, context=context) as server:
    server.login(sender_email, password)
    server.sendmail(sender_email, receiver_email, text)

The MIMEultipart() message accepts parameters in the form of RFC5233-style key/value pairs, which are stored in a dictionary and passed to the .add_header method of the Message base class.

Check out the documentation for Python’s email.mime module to learn more about using MIME classes.

Sending Multiple Personalized Emails

Imagine you want to send emails to members of your organization, to remind them to pay their contribution fees. Or maybe you want to send students in your class personalized emails with the grades for their recent assignment. These tasks are a breeze in Python.

Make a CSV File With Relevant Personal Info

An easy starting point for sending multiple personalized emails is to create a CSV (comma-separated values) file that contains all the required personal information. (Make sure not to share other people’s private information without their consent.) A CSV file can be thought of as a simple table, where the first line often contains the column headers.

Below are the contents of the file contacts_file.csv, which I saved in the same folder as my Python code. It contains the names, addresses, and grades for a set of fictional people. I used my+modifier@gmail.com constructions to make sure all emails end up in my own inbox, which in this example is my@gmail.com:

name,email,grade
Ron Obvious,my+ovious@gmail.com,B+
Killer Rabbit of Caerbannog,my+rabbit@gmail.com,A
Brian Cohen,my+brian@gmail.com,C

When creating a CSV file, make sure to separate your values by a comma, without any surrounding whitespaces.

Loop Over Rows to Send Multiple Emails

The code example below shows you how to open a CSV file and loop over its lines of content (skipping the header row). To make sure that the code works correctly before you send emails to all your contacts, I’ve printed Sending email to ... for each contact, which we can later replace with functionality that actually sends out emails:

import csv

with open("contacts_file.csv") as file:
    reader = csv.reader(file)
    next(reader)  # Skip header row
    for name, email, grade in reader:
        print(f"Sending email to {name}")
        # Send email here

In the example above, using with open(filename) as file:makes sure that your file closes at the end of the code block. csv.reader() makes it easy to read a CSV file line by line and extract its values. The next(reader) line skips the header row, so that the following line for name, email, grade in reader: splits subsequent rows at each comma, and stores the resulting values in the strings name, email and grade for the current contact.

If the values in your CSV file contain whitespaces on either or both sides, you can remove them using the .strip() method.

Personalized Content

You can put personalized content in a message by using str.format() to fill in curly-bracket placeholders. For example, "hi {name}, you {result} your assignment".format(name="John", result="passed") will give you "hi John, you passed your assignment".

As of Python 3.6, string formatting can be done more elegantly using f-strings, but these require the placeholders to be defined before the f-string itself. In order to define the email message at the beginning of the script, and fill in placeholders for each contact when looping over the CSV file, the older .format() method is used.

With this in mind, you can set up a general message body, with placeholders that can be tailored to individuals.

Code Example

The following code example lets you send personalized emails to multiple contacts. It loops over a CSV file with name,email,grade for each contact, as in the example above.

The general message is defined in the beginning of the script, and for each contact in the CSV file its {name} and {grade} placeholders are filled in, and a personalized email is sent out through a secure connection with the Gmail server, as you saw before:

import csv, smtplib, ssl

message = """Subject: Your grade

Hi {name}, your grade is {grade}"""
from_address = "my@gmail.com"
password = input("Type your password and press enter: ")

context = ssl.create_default_context()
with smtplib.SMTP_SSL("smtp.gmail.com", 465, context=context) as server:
    server.login(from_address, password)
    with open("contacts_file.csv") as file:
        reader = csv.reader(file)
        next(reader)  # Skip header row
        for name, email, grade in reader:
            server.sendmail(
                from_address,
                email,
                message.format(name=name,grade=grade),
            )

Yagmail

There are multiple libraries designed to make sending emails easier, such as Envelopes, Flanker and Yagmail. Yagmail is designed to work specifically with Gmail, and it greatly simplifies the process of sending emails through a friendly API, as you can see in the code example below:

import yagmail

receiver = "your@gmail.com"
body = "Hello there from Yagmail"
filename = "document.pdf"

yag = yagmail.SMTP("my@gmail.com")
yag.send(
    to=receiver,
    subject="Yagmail test with attachment",
    contents=body, 
    attachments=filename,
)

This code example sends an email with a PDF attachment in a fraction of the lines needed for our example using email and smtplib.

When setting up Yagmail, you can add your Gmail validations to the keyring of your OS, as described in the documentation. If you don’t do this, Yagmail will prompt you to enter your password when required and store it in the keyring automatically.

Transactional Email Services

If you plan to send a large volume of emails, want to see email statistics, and want to ensure reliable delivery, it may be worth looking into transactional email services. Although all of the following services have paid plans for sending large volumes of emails, they also come with a free plan so you can try them out. Some of these free plans are valid indefinitely and may be sufficient for your email needs.

Below is an overview of the free plans for some of the major transactional email services. Clicking on the provider name will take you to the pricing section of their website.

Provider	Free plan
Sendgrid	40,000 emails for your first 30 days, then 100/day
Sendinblue	300 emails/day
Mailgun	First 10,000 emails free
Mailjet	200 emails/day
Amazon SES	62,000 emails/month

You can run a Google search to see which provider best fits your needs, or just try out a few of the free plans to see which API you like working with most.

Sendgrid Code Example

Here’s a code example for sending emails with Sendgrid to give you a flavor of how to use a transactional email service with Python:

import os
import sendgrid
from sendgrid.helpers.mail import Content, Email, Mail

sg = sendgrid.SendGridAPIClient(
    apikey=os.environ.get("SENDGRID_API_KEY")
)
from_email = Email("my@gmail.com")
to_email = Email("your@gmail.com")
subject = "A test email from Sendgrid"
content = Content(
    "text/plain", "Here's a test email sent through Python"
)
mail = Mail(from_email, subject, to_email, content)
response = sg.client.mail.send.post(request_body=mail.get())

# The statements below can be included for debugging purposes
print(response.status_code)
print(response.body)
print(response.headers)

To run this code, you must first:

Sign up for a (free) Sendgrid account
Request an API key for user validation
Add your API key by typing setx SENDGRID_API_KEY "YOUR_API_KEY" in Command Prompt (to store this API key permanently) or set SENDGRID_API_KEY YOUR_API_KEY to store it only for the current client session

More information on how to set up Sendgrid for Mac and Windows can be found in the repository’s README on Github.

Conclusion

You can now start a secure SMTP connection and send multiple personalized emails to the people in your contacts list!

You’ve learned how to send an HTML email with a plain-text alternative and attach files to your emails. The Yagmail package simplifies all these tasks when you’re using a Gmail account. If you plan to send large volumes of email, it is worth looking into transactional email services.

Enjoy sending emails with Python, and remember: no spam please!

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Building Serverless Python Apps Using AWS Chalice

2018-12-03T14:00:00+00:00

Shipping a web application usually involves having your code up and running on single or multiple servers. In this model, you end up setting up processes for monitoring, provisioning, and scaling your servers up or down. Although this seems to work well, having all the logistics around a web application handled in an automated manner reduces a lot of manual overhead. Enter Serverless.

With Serverless Architecture, you don’t manage servers. Instead, you only need to ship the code or the executable package to the platform that executes it. It’s not really serverless. The servers do exist, but the developer doesn’t need to worry about them.

AWS introduced Lambda Services, a platform that enables developers to simply have their code executed in a particular runtime environment. To make the platform easy to use, many communities have come up with some really good frameworks around it in order to make the serverless apps a working solution.

By the end of this tutorial, you’ll be able to:

Discuss the benefits of a serverless architecture
Explore Chalice, a Python serverless framework
Build a full blown serverless app for a real world use case
Deploy to Amazon Web Services (AWS) Lambda
Compare Pure and Lambda functions

Free Bonus: 5 Thoughts On Python Mastery, a free course for Python developers that shows you the roadmap and the mindset you'll need to take your Python skills to the next level.

Getting Started With AWS Chalice

Chalice, a Python Serverless Microframework developed by AWS, enables you to quickly spin up and deploy a working serverless app that scales up and down on its own as required using AWS Lambda.

Why Chalice?

For Python developers accustomed to the Flask web framework, Chalice should be a breeze in terms of building and shipping your first app. Highly inspired by Flask, Chalice keeps it pretty minimalist in terms of defining what the service should be like and finally making an executable package of the same.

Enough theory! Let’s start with a basic hello-world app and kick-start our serverless journey.

Project Setup

Before diving into Chalice, you’ll set up a working environment on your local machine, which will set you up for the rest of the tutorial.

First, create and activate a virtual environment and install Chalice:

$ python3.6 -m venv env
$ source env/bin/activate
(env)$ pip install chalice

Follow our comprehensive guide on the Pipenv packaging tool.

Note: Chalice comes with a user-friendly CLI that makes it easy to play around with your serverless app.

Now that you have Chalice installed on your virtual environment, let’s use the Chalice CLI to generate some boilerplate code:

(env)$ chalice new-project

Enter the name of the project when prompted and hit return. A new directory is created with that name:

<project-name>/
|
├── .chalice/
│   └── config.json
|
├── .gitignore
├── app.py
└── requirements.txt

See how minimalist the Chalice codebase is. A .chalice directory, app.py, and requirements.txt is all that it requires to have a serverless app up and running. Let’s quickly run the app on our local machine.

Chalice CLI consists of really great utility functions allowing you to perform a number of operations from running locally to deploying in a Lambda environment.

Build and Run Locally

You can simulate the app by running it locally using the local utility of Chalice:

(env)$ chalice local
Serving on 127.0.0.1:8000

By default, Chalice runs on port 8000. We can now check the index route by making a curl request to http://localhost:8000/:

$ curl -X GET http://localhost:8000/
{"hello": "world"}

Now if we look at app.py, we can appreciate the simplicity with which Chalice allows you to build a serverless service. All the complex stuff is handled by the decorators:

from chalice import Chalice
app = Chalice(app_name='serverless-sms-service')

@app.route('/')
def index():
    return {'hello': 'world'}

Note: We haven’t named our app hello-world, as we will build our SMS service on the same app.

Now, let’s move on to deploying our app on the AWS Lambda.

Deploy on AWS Lambda

Chalice makes deploying your serverless app completely effortless. Using the deploy utility, you can simply instruct Chalice to deploy and create a Lambda function that can be accessible via a REST API.

Before we begin deployment, we need to make sure we have our AWS credentials in place, usually located at ~/.aws/config. The contents of the file look as follows:

[default]
aws_access_key_id=<your-access-key-id>
aws_secret_access_key=<your-secret-access-key>
region=<your-region>

With AWS credentials in place, let’s begin our deployment process with just a single command:

(env)$ chalice deploy
Creating deployment package.
Updating policy for IAM role: hello-world-dev
Creating lambda function: hello-world-dev
Creating Rest API
Resources deployed:
  - Lambda ARN: arn:aws:lambda:ap-south-1:679337104153:function:hello-world-dev
  - Rest API URL: https://fqcdyzvytc.execute-api.ap-south-1.amazonaws.com/api/

Note: The generated ARN and API URL in the above snippet will vary from user to user.

Wow! Yes, it is really this easy to get your serverless app up and running. To verify simply make a curl request on the generated Rest API URL:

$ curl -X GET https://fqcdyzvytc.execute-api.ap-south-1.amazonaws.com/api/
{"hello": "world"}

Typically, this is all that you need to get your serverless app up and running. You can also go to your AWS console and see the Lambda function created under the Lambda service section. Each Lambda service has a unique REST API endpoint that can be consumed in any web application.

Next, you will begin building your Serverless SMS Sender service using Twilio as an SMS service provider.

Building a Serverless SMS Sender Service

With a basic hello-world app deployed, let’s move on to building a more real-world application that can be used along with everyday web apps. In this section, you’ll build a completely serverless SMS-sending app that can be plugged into any system and work as expected as long as the input parameters are correct.

In order to send SMS, we will be using Twilio, a developer-friendly SMS service. Before we begin using Twilio, we need to take care of a few prerequisites:

Create an account and acquire ACCOUNT_SID and AUTH_TOKEN.
Get a mobile phone number, which is available for free at Twilio for minor testing stuff.
Install the twilio package in our virtual environment using pip install twilio.

With all the above prerequisites checked, you can start building your SMS service client using Twilio’s Python library. Let’s begin by cloning the repository and creating a new feature branch:

$ git clone <project-url>
$ cd <project-dir>
$ git checkout tags/1.0 -b twilio-support

Now make the following changes to app.py to evolve it from a simple hello-world app to enable support for Twilio service too.

First, let’s include all the import statements:

from os import environ as env

# 3rd party imports
from chalice import Chalice, Response
from twilio.rest import Client
from twilio.base.exceptions import TwilioRestException

# Twilio Config
ACCOUNT_SID = env.get('ACCOUNT_SID')
AUTH_TOKEN = env.get('AUTH_TOKEN')
FROM_NUMBER = env.get('FROM_NUMBER')
TO_NUMBER = env.get('TO_NUMBER')

Next, you’ll encapsulate the Twilio API and use it to send SMS:

app = Chalice(app_name='sms-shooter')

# Create a Twilio client using account_sid and auth token
tw_client = Client(ACCOUNT_SID, AUTH_TOKEN)

@app.route('/service/sms/send', methods=['POST'])
def send_sms():
    request_body = app.current_request.json_body
    if request_body:
        try:
            msg = tw_client.messages.create(
                from_=FROM_NUMBER,
                body=request_body['msg'],
                to=TO_NUMBER)

            if msg.sid:
                return Response(status_code=201,
                                headers={'Content-Type': 'application/json'},
                                body={'status': 'success',
                                      'data': msg.sid,
                                      'message': 'SMS successfully sent'})
            else:
                return Response(status_code=200,
                                headers={'Content-Type': 'application/json'},
                                body={'status': 'failure',
                                      'message': 'Please try again!!!'})
        except TwilioRestException as exc:
            return Response(status_code=400,
                            headers={'Content-Type': 'application/json'},
                            body={'status': 'failure',
                                  'message': exc.msg})

In the above snippet, you simply create a Twilio client object using ACCOUNT_SID and AUTH_TOKEN and use it to send messages under the send_sms view. send_sms is a bare bones function that uses the Twilio client’s API to send the SMS to the specified destination. Before proceeding further, let’s give it a try and run it on our local machine.

Build and Run Locally

Now you can run your app on your machine using the local utility and verify that everything is working fine:

(env)$ chalice local

Now make a curl POST request to http://localhost:8000/service/sms/send with a specific payload and test the app locally:

$ curl -H "Content-Type: application/json" -X POST -d '{"msg": "hey mate!!!"}' http://localhost:8000/service/sms/send

The above request responds as follows:

{
  "status": "success",
  "data": "SM60f11033de4f4e39b1c193025bcd5cd8",
  "message": "SMS successfully sent"
}

The response indicates that the message was successfully sent. Now, let’s move on to deploying the app on AWS Lambda.

Deploy on AWS Lambda

As suggested in the previous deployment section, you just need to issue the following command:

(env)$ chalice deploy
Creating deployment package.
Updating policy for IAM role: sms-shooter-dev
Creating lambda function: sms-shooter-dev
Creating Rest API
Resources deployed:
  - Lambda ARN: arn:aws:lambda:ap-south-1:679337104153:function:sms-shooter-dev
  - Rest API URL: https://qtvndnjdyc.execute-api.ap-south-1.amazonaws.com/api/

Note: The above command succeeds, and you have your API URL in the output as expected. Now on testing the URL, the API throws an error message. What went wrong?

As per AWS Lambda logs, twilio package is not found or installed, so you need to tell the Lambda service to install the dependencies. To do so, you need to add twilio as a dependency to requirements.txt:

twilio==6.18.1

Other packages such as Chalice and its dependencies should not be included in requirements.txt, as they are not a part of Python’s WSGI runtime. Instead, we should maintain a requirements-dev.txt, which is applicable to only the development environment and contains all Chalice-related dependencies. To learn more, check out this GitHub issue.

Once all the package dependencies are sorted, you need to make sure all the environment variables are also shipped along and set correctly during the Lambda runtime. To do so, you have to add all the environment variables in .chalice/config.json in the following manner:

{
  "version": "2.0",
  "app_name": "sms-shooter",
  "stages": {
    "dev": {
      "api_gateway_stage": "api",
      "environment_variables": {
          "ACCOUNT_SID": "<your-account-sid>",
          "AUTH_TOKEN": "<your-auth-token>",
          "FROM_NUMBER": "<source-number>",
          "TO_NUMBER": "<destination-number>"
      }
    }
  }
}

Now we’re good to deploy:

Creating deployment package.
Updating policy for IAM role: sms-shooter-dev
Updating lambda function: sms-shooter-dev
Updating rest API
Resources deployed:
  - Lambda ARN: arn:aws:lambda:ap-south-1:679337104153:function:sms-shooter-dev
  - Rest API URL: https://fqcdyzvytc.execute-api.ap-south-1.amazonaws.com/api/

Do a sanity check by making a curl request to the generated API endpoint:

$ curl -H "Content-Type: application/json" -X POST -d '{"msg": "hey mate!!!"}' https://fqcdyzvytc.execute-api.ap-south-1.amazonaws.com/api/service/sms/send

The above request responds as expected:

{
  "status": "success",
  "data": "SM60f11033de4f4e39b1c193025bcd5cd8",
  "message": "SMS successfully sent"
}

Now, you have a completely serverless SMS sending service up and running. With the front end of this service being a REST API, it can be used in other applications as a plug-and-play feature that is scalable, secure, and reliable.

Refactoring

Finally, we will refactor our SMS app to not contain all the business logic in app.py completely. Instead, we will follow the Chalice prescribed best practices and abstract the business logic under the chalicelib/ directory.

Let’s begin by creating a new branch:

$ git checkout tags/2.0 -b sms-app-refactor

First, create a new directory in the root directory of the project named chalicelib/ and create a new file named sms.py:

(env)$ mkdir chalicelib
(env)$ touch chalicelib/sms.py

Update the above created chalicelib/sms.py with the SMS sending logic by abstracting things from app.py:

from os import environ as env
from twilio.rest import Client

# Twilio Config
ACCOUNT_SID = env.get('ACCOUNT_SID')
AUTH_TOKEN = env.get('AUTH_TOKEN')
FROM_NUMBER = env.get('FROM_NUMBER')
TO_NUMBER = env.get('TO_NUMBER')

# Create a twilio client using account_sid and auth token
tw_client = Client(ACCOUNT_SID, AUTH_TOKEN)


def send(payload_params=None):
    """ send sms to the specified number """
    msg = tw_client.messages.create(
        from_=FROM_NUMBER,
        body=payload_params['msg'],
        to=TO_NUMBER)

    if msg.sid:
        return msg

The above snippet only accepts the input params and responds as required. Now to make this work, we need to make changes to app.py as well:

# Core imports
from chalice import Chalice, Response
from twilio.base.exceptions import TwilioRestException

# App level imports
from chalicelib import sms

app = Chalice(app_name='sms-shooter')


@app.route('/')
def index():
    return {'hello': 'world'}


@app.route('/service/sms/send', methods=['POST'])
def send_sms():
    request_body = app.current_request.json_body
    if request_body:
        try:
            resp = sms.send(request_body)
            if resp:
                return Response(status_code=201,
                                headers={'Content-Type': 'application/json'},
                                body={'status': 'success',
                                      'data': resp.sid,
                                      'message': 'SMS successfully sent'})
            else:
                return Response(status_code=200,
                                headers={'Content-Type': 'application/json'},
                                body={'status': 'failure',
                                      'message': 'Please try again!!!'})
        except TwilioRestException as exc:
            return Response(status_code=400,
                            headers={'Content-Type': 'application/json'},
                            body={'status': 'failure',
                                  'message': exc.msg})

In the above snippet, all the SMS sending logic is invoked from the chalicelib.sms module, making the view layer a lot cleaner in terms of readability. This abstraction lets you add much more complex business logic and customize the functionality as required.

Sanity Check

After refactoring our code, let’s ensure it is running as expected.

Build and Run Locally

Run the app once again using the local utility:

(env)$ chalice local

Make a curl request and verify. Once that’s done, move on to deployment.

Deploy on AWS Lambda

Once you are sure everything is working as expected, you can now finally deploy your app:

(env)$ chalice deploy

As usual, the command executes successfully and you can verify the endpoint.

Conclusion

You now know how to do the following:

Build a serverless application using AWS Chalice in accordance with best practices
Deploy your working app on the Lambda runtime environment

Lambda services under the hood are analogous to pure functions, which have a certain behavior on a set of input/output. Developing precise Lambda services allows for better testing, readability, and atomicity. Since Chalice is a minimalist framework, you can just focus on the business logic, and the rest is taken care of, from deployment to IAM policy generation. This is all with just a single command deployment!

Moreover, Lambda services are mostly focused on heavy CPU bound processing and scale in a self-governed manner, as per the number of requests in a unit of time. Using serverless architecture allows your codebase to be more like SOA (Service Oriented Architecture). Using AWS’s other products in their ecosystem that plug in well with Lambda functions is even more powerful.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python Community Interview With Emily Morehouse

2018-11-28T14:00:00+00:00

I’m very pleased to be joined this week by Emily Morehouse.

Emily is one of the newest additions to the CPython core developer team, and the founder and director of engineering of Cuttlesoft. Emily and I talk about the recent CPython core developer sprint and the fact that she completed three majors in college at the same time! We’ll also get into her passion for compilers and abstract syntax trees.

Ricky: Let’s start with the obvious: how did you get into programming, and when did you start using Python?

Emily: My path to programming started with falling in love with Enigma machines. Really, I stumbled into programming during college. I had recently switched one of my majors from Biochemistry to Criminology, and the department had just launched a Computer Criminology program.

I was encouraged to try out the Intro To Programming course to see how I liked it. One of our final projects was building an Enigma machine simulator (in C++, mind you), and I was hooked. I decided to add a third major to take on a full Computer Science degree (more on that later!).

Since the CS program was highly theoretical and focused on languages like C and C++, I started to find ways outside of coursework to learn different things. I picked up Python working on web scrapers on the weekends and was eventually hired as a researcher where we used Python to scrape and analyze public data from various sites.

For me, programming spans this wide range of challenging logic and technical problems to more abstract concepts of how humans think and interact with machines and how technology can enhance our daily lives. It fills a gap between academics and art that I didn’t know I needed to, or could, fill.

Ricky: As you’ve already alluded to, you attended Florida State University, where you completed your CS degree. And a degree in Criminology. And another one in Theater… Did you ever sleep? One degree is hard, but three at once? I’m really curious to know your secrets and any time management hacks for studying and learning to code when you have so much else going on.

Emily: I definitely did not sleep much. On top of all of my schoolwork, I worked a nearly full-time job and even worked as an overnight manager at our local coffee shop while still participating in theater rehearsals and performances.

I was able to get a research position in the CS department to eliminate some of that pressure. I was lucky to have started college with a lot of credits and tested out of a few courses, so I was technically already a year ahead, which gave me more freedom to try out courses like Programming.

It’s all very much how I was raised. From a very young age, I knew that my day started around 7 a.m. I went straight from school to rehearsals and dance classes, then had to do homework until I fell asleep. I had to learn how to retain information and figure things out quickly—and I had to stay organized, so I made a lot of lists.

I’ve asked my parents how I came to be this way, and they just shrug! I’ve always felt very in control of how I spend my time to ensure it’s what I want to be doing, and I think that’s important when staying so busy. You have to want to be doing everything, or else things will fall by the wayside.

I definitely suggest finding a manner of keeping to-do lists and prioritizing your time. I use an app called Bear (like a simplified Evernote, but with programmer-friendly themes and markdown support) along with a lot of task prioritization.

I also figured out that I learn things quickly by writing them down multiple times. I used this method to memorize lines for shows. I’d white-out my lines then go back and physically write them down from memory on a separate piece of paper, rinse, and repeat. I got to the point where if I wrote something down 1 to 2 times, it’d stick.

Ricky: You are the co-founder and director of engineering at Cuttlesoft. It looks as if you started the company before finishing college. What was your motivation for starting your own business instead of applying for junior software developer jobs straight out of college?

Emily: Cuttlesoft was a matter of circumstance. I never imagined I’d run my own company, especially not alongside my now-husband Frank. I was in a weird timing limbo where I’d finished my undergraduate degrees earlier than expected which meant that I missed all of the grad school deadlines.

FSU agreed to let me jump in and start my Masters there, and my intention was to stay for a year then transfer elsewhere where I could continue working with parsers, compilers, and formal verification. I was also getting recruited by huge tech companies, and I was a bit enamored with the idea of living in San Francisco or Boston. (I’d only ever lived in Florida at the time.)

But then Frank and I had found our way into this budding entrepreneurship ecosystem in Tallahassee. We met a few people who became great mentors, and before we could even get our Is dotted, we had our first couple of clients. I thought, “Why do I want to leave all of these people who are invested in my future and success to go be one of the thousands somewhere else?”

I figured that I should take the chance on starting something of my own and continuing on this rapid growth path. I knew I’d learn a lot more in a shorter amount of time than I would almost anywhere else. So I dropped out of graduate school after my first semester and put all of my time into Cuttlesoft.

Looking back, I can’t imagine a different path for me. Soon after I turned down those job offers, Susan Fowler’s story came to light. I couldn’t help but think, “That could have been me.” I truly believe that a company’s culture is top-down, and I’m grateful to get to contribute to a company where I can make a huge impact in a positive manner.

Ricky: This year, you got to fulfill a dream and speak at PyCon, with your talk titled The AST and Me. I’ll admit, some of it went over my head, but I’m still learning. I got the impression that language internals fascinate you. What advice would you give to someone who is at the start of their coding journey and wants to know more about how the sausage is made? What resources would you recommend?

Emily: Yes! I was the weird kid in university who loved the classes that most everyone else hated (Programming Languages, Compilers, Theory of Computation…). I would spend hours drawing out non-deterministic finite automata and state machines for my course notes.

I’m a huge fan of Seven Languages in Seven Weeks: A Pragmatic Guide to Learning Programming Languages by Bruce A. Tate to get a feel for different programming language paradigms. The Dragon Book (Compilers: Principles, Techniques, and Tools) is a classic and is the backbone of so much we still use today. (Python’s compiler is based on this.) Philip Guo’s video series on CPython internals is also awesome and helped me in my journey diving into how Python works under the hood.

Ricky: Huge congratulations are in order, as you have just been promoted to a CPython core developer! You must be so thrilled. How was the initiation at the recent CPython sprints? Anything exciting to share or any stories to tell? Don’t worry, we can keep a secret…

Emily: Thank you! The CPython Sprint was a lot of fun. It’s rare that we get so many core developers in the same room working together. We’re all incredibly grateful for the PSF and this year’s sprint sponsor, Microsoft, for supporting CPython.

I was able to attend sprints and Language Summits at PyCons past and had the chance to get to know a lot of the group previously, so this sprint felt surprisingly normal, but it was super cool to see and work with everyone in person.

I spent most of the sprint implementing PEP 572, the (in)famous assignment expressions PEP, with Guido’s guidance. No matter which side of the fence you fell on with assignment expressions (or I as lovingly now call it, the walrus operator), it’s been incredibly cool to add new syntax to the language and deep dive into the internals to get variable scoping to work as intended. It will be in the alpha versions of 3.8 early next year, so keep an eye out!

One of my favorite parts of the sprint was getting to know more about the history of CPython. Since the beginning of my path in core development, I’ve found that it’s really interesting to hear the stories of how others became core developers, so I pose that question to everyone I can.

Understanding everyone’s journey and motivations for devoting so much of their time and energy to a project (especially those who have been involved since the very early days) is an important step to understanding how to continue growing the group and increasing diversity.

Ricky: Now for my last question: what other hobbies and interests do you have, aside from Python? Any you’d like to share and/or plug?

Emily: I try to take advantage of everything Colorado has to offer in my spare time—coming from Florida, I’m still totally enamored with the Rocky Mountains and love hiking. Denver is also a great foodie city.

When I make the time for it, I also really enjoy yoga, reading, listening to podcasts, playing video games (though I’m still slowly working through the most recent God of War), and trying to keep my houseplants alive. I also enjoy spending time with my husband and our dog—they’re my world.

Thank you, Emily, for joining me this week. You can follow Emily’s work on Twitter or Github. Find out more about her company, Cuttlesoft, here.

If there’s someone you’d like me to interview in the future, reach out to me in the comments below, or send me a message on Twitter.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Continuous Integration With Python: An Introduction

2018-11-26T14:00:00+00:00

When writing code on your own, the only priority is making it work. However, working in a team of professional software developers brings a plethora of challenges. One of those challenges is coordinating many people working on the same code.

How do professional teams make dozens of changes per day while making sure everyone is coordinated and nothing is broken? Enter continuous integration!

In this tutorial you’ll:

Learn the core concepts behind continuous integration
Understand the benefits of continuous integration
Set up a basic continuous integration system
Create a simple Python example and connect it to the continuous integration system

Free Bonus: 5 Thoughts On Python Mastery, a free course for Python developers that shows you the roadmap and the mindset you'll need to take your Python skills to the next level.

What Is Continuous Integration?

Continuous integration (CI) is the practice of frequently building and testing each change done to your code automatically and as early as possible. Prolific developer and author Martin Fowler defines CI as follows:

“Continuous Integration is a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily - leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible.” (Source)

Let’s unpack this.

Programming is iterative. The source code lives in a repository that is shared by all members of the team. If you want to work on that product, you must obtain a copy. You will make changes, test them, and integrate them back into the main repo. Rinse and repeat.

Not so long ago, these integrations were big and weeks (or months) apart, causing headaches, wasting time, and losing money. Armed with experience, developers started making minor changes and integrating them more frequently. This reduces the chances of introducing conflicts that you need to resolve later.

After every integration, you need to build the source code. Building means transforming your high-level code into a format your computer knows how to run. Finally, the result is systematically tested to ensure your changes did not introduce errors.

Why Should I Care?

On a personal level, continuous integration is really about how you and your colleagues spend your time.

Using CI, you’ll spend less time:

Worrying about introducing a bug every time you make changes
Fixing the mess someone else made so you can integrate your code
Making sure the code works on every machine, operating system, and browser

Conversely, you’ll spend more time:

Solving interesting problems
Writing awesome code with your team
Co-creating amazing products that provide value to users

How does that sound?

On a team level, it allows for a better engineering culture, where you deliver value early and often. Collaboration is encouraged, and bugs are caught much sooner. Continuous integration will:

Make you and your team faster
Give you confidence that you’re building stable software with fewer bugs
Ensure that your product works on other machines, not just your laptop
Eliminate a lot of tedious overhead and let you focus on what matters
Reduce the time spent resolving conflicts (when different people modify the same code)

Core Concepts

There are several key ideas and practices that you need to understand to work effectively with continuous integration. Also, there might be some words and phrases you aren’t familiar with but are used often when you’re talking about CI. This chapter will introduce you to these concepts and the jargon that comes with them.

Single Source Repository

If you are collaborating with others on a single code base, it’s typical to have a shared repository of source code. Every developer working on the project creates a local copy and makes changes. Once they are satisfied with the changes, they merge them back into the central repository.

It has become a standard to use version control systems (VCS) like Git to handle this workflow for you. Teams typically use an external service to host their source code and handle all the moving parts. The most popular are GitHub, BitBucket, and GitLab.

Git allows you to create multiple branches of a repository. Each branch is an independent copy of the source code and can be modified without affecting other branches. This is an essential feature, and most teams have a mainline branch (often called a master branch) that represents the current state of the project.

If you want to add or modify code, you should create a copy of the main branch and work in your new, development branch. Once you are done, merge those changes back into the master branch.

Git branching

Version control holds more than just code. Documentation and test scripts are usually stored along with the source code. Some programs look for external files used to configure their parameters and initial settings. Other applications need a database schema. All these files should go into your repository.

If you have never used Git or need a refresher, check out our Introduction to Git and GitHub for Python Developers.

Automating the Build

As previously mentioned, building your code means taking the raw source code, and everything necessary for its execution, and translating it into a format that computers can run directly. Python is an interpreted language, so its “build” mainly revolves around test execution rather than compilation.

Running those steps manually after every small change is tedious and takes valuable time and attention from the actual problem-solving you’re trying to do. A big part of continuous integration is automating that process and moving it out of sight (and out of mind).

What does that mean for Python? Think about a more complicated piece of code you have written. If you used a library, package, or framework that doesn’t come with the Python standard library (think anything you needed to install with pip or conda), Python needs to know about that, so the program knows where to look when it finds commands that it doesn’t recognize.

You store a list of those packages in requirements.txt or a Pipfile. These are the dependencies of your code and are necessary for a successful build.

You will often hear the phrase “breaking the build.” When you break the build, it means you introduced a change that rendered the final product unusable. Don’t worry. It happens to everyone, even battle-hardened senior developers. You want to avoid this primarily because it will block everyone else from working.

The whole point of CI is to have everyone working on a known stable base. If they clone a repository that is breaking the build, they will work with a broken version of the code and won’t be able to introduce or test their changes. When you break the build, the top priority is fixing it so everyone can resume work.

Introducing a breaking change to the master branch

When the build is automated, you are encouraged to commit frequently, usually multiple times per day. It allows people to quickly find out about changes and notice if there’s a conflict between two developers. If there are numerous small changes instead of a few massive updates, it’s much easier to locate where the error originated. It will also encourage you to break your work down into smaller chunks, which is easier to track and test.

Automated Testing

Since everyone is committing changes multiple times per day, it’s important to know that your change didn’t break anything else in the code or introduce bugs. In many companies, testing is now a responsibility of every developer. If you write code, you should write tests. At a bare minimum, you should cover every new function with a unit test.

Running tests automatically, with every change committed, is a great way to catch bugs. A failing test automatically causes the build to fail. It will draw your attention to the problems revealed by testing, and the failed build will make you fix the bug you introduced. Tests don’t guarantee that your code is free of bugs, but it does guard against a lot of careless changes.

Automating test execution gives you some peace of mind because you know the server will test your code every time you commit, even if you forgot to do it locally.

Using an External Continuous Integration Service

If something works on your computer, will it work on every computer? Probably not. It’s a cliché excuse and a sort of inside joke among developers to say, “Well, it worked on my machine!” Making the code work locally is not the end of your responsibility.

To tackle this problem, most companies use an external service to handle integration, much like using GitHub for hosting your source code repository. External services have servers where they build code and run tests. They act as monitors for your repository and stop anyone from merging to the master branch if their changes break the build.

Merging changes triggers the CI server

There are many such services out there, with various features and pricing. Most have a free tier so that you can experiment with one of your repositories. You will use a service called CircleCI in an example later in the tutorial.

Testing in a Staging Environment

A production environment is where your software will ultimately run. Even after successfully building and testing your application, you can’t be sure that your code will work on the target computer. That’s why teams deploy the final product in an environment that mimics the production environment. Once you are sure everything works, the application is deployed in the production environment.

Note: This step is more relevant to application code than library code. Any Python libraries you write still need to be tested on a build server, to ensure they work in environments different from your local computer.

You will hear people talking about this clone of the production environment using terms like development environment, staging environment, or testing environment. It’s common to use abbreviations like DEV for the development environment and PROD for the production environment.

The development environment should replicate production conditions as closely as possible. This setup is often called DEV/PROD parity. Keep the environment on your local computer as similar as possible to the DEV and PROD environments to minimize anomalies when deploying applications.

Test in a clone of the production environment

We mention this to introduce you to the vocabulary, but continuously deploying software to DEV and PROD is a whole other topic. The process is called, unsurprisingly, continuous deployment (CD). You can find more resources about it in the Next Steps section of this article.

Your Turn!

The best way to learn is by doing. You now understand all the essential practices of continuous integration, so it’s time to get your hands dirty and create the whole chain of steps necessary to use CI. This chain is often called a CI pipeline.

This is a hands-on tutorial, so fire up your editor and get ready to work through these steps as you read!

We assume that you know the basics of Python and Git. We will use Github as our hosting service and CircleCI as our external continuous integration service. If you don’t have accounts with these services, go ahead and register. Both of these have free tiers!

Problem Definition

Remember, your focus here is adding a new tool to your utility belt, continuous integration. For this example, the Python code itself will be straightforward. You want to spend the bulk of your time internalizing the steps of building a pipeline, instead of writing complicated code.

Imagine your team is working on a simple calculator app. Your task is to write a library of basic mathematical functions: addition, subtraction, multiplication, and division. You don’t care about the actual application, because that’s what your peers will be developing, using functions from your library.

Create a Repo

Log in to your GitHub account, create a new repository and call it CalculatorLibrary. Add a README and .gitignore, then clone the repository to your local machine. If you need more help with this process, have a look at GitHub’s walkthrough on creating a new repository.

Set Up a Working Environment

For others (and the CI server) to replicate your working conditions, you need to set up an environment. Create a virtual environment somewhere outside your repo and activate it:

$ # Create virtual environment
$ python3 -m venv calculator

$ # Activate virtual environment (Mac and Linux)
$ . calculator/bin/activate

The previous commands work on macOS and Linux. If you are a Windows user, check the Platforms table in the official documentation. This will create a directory that contains a Python installation and tell the interpreter to use it. Now we can install packages knowing that it will not influence your system’s default Python installation.

Write a Simple Python Example

Create a new file called calculator.py in the top-level directory of your repository, and copy the following code:

"""
Calculator library containing basic math operations.
"""

def add(first_term, second_term):
    return first_term + second_term

def subtract(first_term, second_term):
    return first_term - second_term

This is a bare-bones example containing two of the four functions we will be writing. Once we have our CI pipeline up and running, you will add the remaining two functions.

Go ahead and commit those changes:

$ # Make sure you are in the correct directory
$ cd CalculatorLibrary
$ git add calculator.py
$ git commit -m "Add functions for addition and subtraction"

Your CalculatorLibrary folder should have the following files right now:

CalculatorLibrary/
|
├── .git
├── .gitignore
├── README.md
└── calculator.py

Great, you have completed one part of the required functionality. The next step is adding tests to make sure your code works the way it’s supposed to.

Write Unit Tests

You will test your code in two steps.

The first step involves linting—running a program, called a linter, to analyze code for potential errors. flake8 is commonly used to check if your code conforms to the standard Python coding style. Linting makes sure your code is easy to read for the rest of the Python community.

The second step is unit testing. A unit test is designed to check a single function, or unit, of code. Python comes with a standard unit testing library, but other libraries exist and are very popular. This example uses pytest.

A standard practice that goes hand in hand with testing is calculating code coverage. Code coverage is the percentage of source code that is “covered” by your tests. pytest has an extension, pytest-cov, that helps you understand your code coverage.

These are external dependencies, and you need to install them:

$ pip install flake8 pytest pytest-cov

These are the only external packages you will use. Make sure to store those dependencies in a requirements.txt file so others can replicate your environment:

$ pip freeze > requirements.txt

To run your linter, execute the following:

$ flake8 --statistics
./calculator.py:3:1: E302 expected 2 blank lines, found 1
./calculator.py:6:1: E302 expected 2 blank lines, found 1
2     E302 expected 2 blank lines, found 1

The --statistics option gives you an overview of how many times a particular error happened. Here we have two PEP 8 violations, because flake8 expects two blank lines before a function definition instead of one. Go ahead and add an empty line before each functions definition. Run flake8 again to check that the error messages no longer appear.

Now it’s time to write the tests. Create a file called test_calculator.py in the top-level directory of your repository and copy the following code:

"""
Unit tests for the calculator library
"""

import calculator


class TestCalculator:

    def test_addition(self):
        assert 4 == calculator.add(2, 2)

    def test_subtraction(self):
        assert 2 == calculator.subtract(4, 2)

These tests make sure that our code works as expected. It is far from extensive because you haven’t tested for potential misuse of your code, but keep it simple for now.

The following command runs your test:

$ pytest -v --cov
collected 2 items

test_calculator.py::TestCalculator::test_addition PASSED [50%]

test_calculator.py::TestCalculator::test_subtraction PASSED [100%]

---------- coverage: platform darwin, python 3.6.6-final-0 -----------
Name                                              Stmts   Miss  Cover
---------------------------------------------------------------------
calculator.py                                         4      0   100%
test_calculator.py                                    6      0   100%
/Users/kristijan.ivancic/code/learn/__init__.py       0      0   100%
---------------------------------------------------------------------
TOTAL                                                10      0   100%

pytest is excellent at test discovery. Because you have a file with the prefix test, pytest knows it will contain unit tests for it to run. The same principles apply to the class and method names inside the file.

The -v flag gives you a nicer output, telling you which tests passed and which failed. In our case, both tests passed. The --cov flag makes sure pytest-cov runs and gives you a code coverage report for calculator.py.

You have completed the preparations. Commit the test file and push all those changes to the master branch:

$ git add test_calculator.py
$ git commit -m "Add unit tests for calculator"
$ git push

At the end of this section, your CalculatorLibrary folder should have the following files:

CalculatorLibrary/
|
├── .git
├── .gitignore
├── README.md
├── calculator.py
├── requirements.txt
└── test_calculator.py

Excellent, both your functions are tested and work correctly.

Connect to CircleCI

At last, you are ready to set up your continuous integration pipeline!

CircleCI needs to know how to run your build and expects that information to be supplied in a particular format. It requires a .circleci folder within your repo and a configuration file inside it. A configuration file contains instructions for all the steps that the build server needs to execute. CircleCI expects this file to be called config.yml.

A .yml file uses a data serialization language, YAML, and it has its own specification. The goal of YAML is to be human readable and to work well with modern programming languages for common, everyday tasks.

In a YAML file, there are three basic ways to represent data:

Mappings (key-value pairs)
Sequences (lists)
Scalars (strings or numbers)

It is very simple to read:

Indentation may be used for structure.
Colons separate key-value pairs.
Dashes are used to create lists.

Create the .circleci folder in your repo and a config.yml file with the following content:

# Python CircleCI 2.0 configuration file
version: 2
jobs:
  build:
    docker:
      - image: circleci/python:3.7

    working_directory: ~/repo

    steps:
      # Step 1: obtain repo from GitHub
      - checkout
      # Step 2: create virtual env and install dependencies
      - run:
          name: install dependencies
          command: |
            python3 -m venv venv
            . venv/bin/activate
            pip install -r requirements.txt
      # Step 3: run linter and tests
      - run:
          name: run tests
          command: |
            . venv/bin/activate
            flake8 --exclude=venv* --statistics
            pytest -v --cov=calculator

Some of these words and concepts might be unfamiliar to you. For example, what is Docker, and what are images? Let’s go back in time a bit.

Remember the problem programmers face when something works on their laptop but nowhere else? Before, developers used to create a program that isolates a part of the computer’s physical resources (memory, hard drive, and so on) and turns them into a virtual machine.

A virtual machine pretends to be a whole computer on its own. It would even have its own operating system. On that operating system, you deploy your application or install your library and test it.

Virtual machines take up a lot of resources, which sparked the invention of containers. The idea is analogous to shipping containers. Before shipping containers were invented, manufacturers had to ship goods in a wide variety of sizes, packaging, and modes (trucks, trains, ships).

By standardizing the shipping container, these goods could be transferred between different shipping methods without any modification. The same idea applies to software containers.

Containers are a lightweight unit of code and its runtime dependencies, packaged in a standardized way, so they can quickly be plugged in and run on the Linux OS. You don’t need to create a whole virtual operating system, as you would with a virtual machine.

Containers only replicate parts of the operating system they need in order to work. This reduces their size and gives them a big performance boost.

Docker is currently the leading container platform, and it’s even able to run Linux containers on Windows and macOS. To create a Docker container, you need a Docker image. Images provide blueprints for containers much like classes provide blueprints for objects. You can read more about Docker in their Get Started guide.

CircleCI maintains pre-built Docker images for several programming languages. In the above configuration file, you have specified a Linux image that has Python already installed. That image will create a container in which everything else happens.

Let’s look at each line of the configuration file in turn:

version: Every config.yml starts with the CircleCI version number, used to issue warnings about breaking changes.
jobs: Jobs represent a single execution of the build and are defined by a collection of steps. If you have only one job, it must be called build.
build: As mentioned before, build is the name of your job. You can have multiple jobs, in which case they need to have unique names.
docker: The steps of a job occur in an environment called an executor. The common executor in CircleCI is a Docker container. It is a cloud-hosted execution environment but other options exist, like a macOS environment.
image: A Docker image is a file used to create a running Docker container. We are using an image that has Python 3.7 preinstalled.
working_directory: Your repository has to be checked out somewhere on the build server. The working directory represents the file path where the repository will be stored.
steps: This key marks the start of a list of steps to be performed by the build server.
checkout: The first step the server needs to do is check the source code out to the working directory. This is performed by a special step called checkout.
run: Executing command-line programs or commands is done inside the command key. The actual shell commands will be nested within.
name: The CircleCI user interface shows you every build step in the form of an expandable section. The title of the section is taken from the value associated with the name key.
command: This key represents the command to run via the shell. The | symbol specifices that what follows is a literal set of commands, one per line, exactly like you’d see in a shell/bash script.

You can read the CircleCI configuration reference document for more information.

Our pipeline is very simple and consists of 3 steps:

Checking out the repository
Installing the dependencies in a virtual environment
Running the linter and tests while inside the virtual environment

We now have everything we need to start our pipeline. Log in to your CircleCI account and click on Add Projects. Find your CalculatorLibrary repo and click Set Up Project. Select Python as your language. Since we already have a config.yml, we can skip the next steps and click Start building.

CircleCI will take you to the execution dashboard for your job. If you followed all the steps correctly, you should see your job succeed.

The final version of your CalculatorLibrary folder should look like this:

CalculatorRepository/
|
├── .circleci
├── .git
├── .gitignore
├── README.md
├── calculator.py
├── requirements.txt
└── test_calculator.py

Congratulations! You have created your first continuous integration pipeline. Now, every time you push to the master branch, a job will be triggered. You can see a list of your current and past jobs by clicking on Jobs in the CircleCI sidebar.

Make Changes

Time to add multiplication to our calculator library.

This time, we will first add a unit test without writing the function. Without the code, the test will fail, which will also fail the CircleCI job. Add the following code to the end of your test_calculator.py:

def test_multiplication(self):
    assert 100 == calculator.multiply(10, 10)

Push the code to the master branch and see the job fail in CircleCI. This shows that continuous integration works and watches your back if you make a mistake.

Now add the code to calculator.py that will make the test pass:

def multiply(first_term, second_term):
    return first_term * second_term

Make sure there are two empty spaces between the multiplication function and the previous one, or else your code will fail the linter check.

The job should be successful this time. This workflow of writing a failing test first and then adding the code to pass the test is called test driven development (TDD). It’s a great way to work because it makes you think about your code structure in advance.

Now try it on your own. Add a test for the division function, see it fail, and write the function to make the test pass.

Notifications

When working on big applications that have a lot of moving parts, it can take a while for the continuous integration job to run. Most teams set up a notification procedure to let them know if one of their jobs fail. They can continue working while waiting for the job to run.

The most popular options are:

Sending an email for each failed build
Sending failure notifications to a Slack channel
Displaying failures on a dashboard visible to everyone

By default, CircleCI should send you an email when a job fails.

Next Steps

You have understood the basics of continuous integration and practiced setting up a pipeline for a simple Python program. This is a big step forward in your journey as a developer. You might be asking yourself, “What now?”

To keep things simple, this tutorial skimmed over some big topics. You can grow your skill set immensely by spending some time going more in-depth into each subject. Here are some topics you can look into further.

Git Workflows

There is much more to Git than what you used here. Each developer team has a workflow tailored to their specific needs. Most of them include branching strategies and something called peer review. They make changes on branches separate from the master branch. When you want to merge those changes with master, other developers must first look at your changes and approve them before you’re allowed to merge.

Note: If you want to learn more about different workflows teams use, have a look at the tutorials on GitHub and BitBucket.

If you want to sharpen your Git skills, we have an article called Advanced Git Tips for Python Developers.

Dependency Management and Virtual Environments

Apart from virtualenv, there are other popular package and environment managers. Some of them deal with just virtual environments, while some handle both package installation and environment management. One of them is Conda:

“Conda is an open source package management system and environment management system that runs on Windows, macOS, and Linux. Conda quickly installs, runs and updates packages and their dependencies. Conda easily creates, saves, loads and switches between environments on your local computer. It was designed for Python programs, but it can package and distribute software for any language.” (Source)

Another option is Pipenv, a younger contender that is rising in popularity among application developers. Pipenv brings together pip and virtualenv into a single tool and uses a Pipfile instead of requirements.txt. Pipfiles offer deterministic environments and more security. This introduction doesn’t do it justice, so check out Pipenv: A Guide to the New Python Packaging Tool.

Testing

Simple unit tests with pytest are only the tip of the iceberg. There’s a whole world out there to explore! Software can be tested on many levels, including integration testing, acceptance testing, regression testing, and so forth. To take your knowledge of testing Python code to the next level, head over to Getting Started With Testing in Python.

Packaging

In this tutorial, you started to build a library of functions for other developers to use in their project. You need to package that library into a format that is easy to distribute and install using, for example pip.

Creating an installable package requires a different layout and some additional files like __init__.py and setup.py. Read Python Application Layouts: A Reference for more information on structuring your code.

To learn how to turn your repository into an installable Python package, read Packaging Python Projects by the Python Packaging Authority.

Continuous Integration

You covered all the basics of CI in this tutorial, using a simple example of Python code. It’s common for the final step of a CI pipeline to create a deployable artifact. An artifact represents a finished, packaged unit of work that is ready to be deployed to users or included in complex products.

For example, to turn your calculator library into a deployable artifact, you would organize it into an installable package. Finally, you would add a step in CircleCI to package the library and store that artifact where other processes can pick it up.

For more complex applications, you can create a workflow to schedule and connect multiple CI jobs into a single execution. Feel free to explore the CircleCI documentation.

Continuous Deployment

You can think of continuous deployment as an extension of CI. Once your code is tested and built into a deployable artifact, it is deployed to production, meaning the live application is updated with your changes. One of the goals is to minimize lead time, the time elapsed between writing a new line of code and putting it in front of users.

Note: To add a bit of confusion to the mix, the acronym CD is not unique. It can also mean Continuous Delivery, which is almost the same as continuous deployment but has a manual verification step between integration and deployment. You can integrate your code at any time but have to push a button to release it to the live application.

Most companies use CI/CD in tandem, so it’s worth your time to learn more about Continuous Delivery/Deployment.

Overview of Continuous Integration Services

You have used CircleCI, one of the most popular continuous integration services. However, this is a big market with a lot of strong contenders. CI products fall into two basic categories: remote and self-hosted services.

Jenkins is the most popular self-hosted solution. It is open-source and flexible, and the community has developed a lot of extensions.

In terms of remote services, there are many popular options like TravisCI, CodeShip, and Semaphore. Big enterprises often have their custom solutions, and they sell them as a service, such as AWS CodePipeline, Microsoft Team Foundation Server, and Oracle’s Hudson.

Which option you choose depends on the platform and features you and your team need. For a more detailed breakdown, have a look at Best CI Software by G2Crowd.

Conclusion

With the knowledge from this tutorial under your belt, you can now answer the following questions:

What is continuous integration?
Why is continuous integration important?
What are the core practices of continuous integration?
How can I set up continuous integration for my Python project?

You have acquired a programming superpower! Understanding the philosophy and practice of continuous integration will make you a valuable member of any team. Awesome work!

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Memory Management in Python

2018-11-21T14:00:00+00:00

Ever wonder how Python handles your data behind the scenes? How are your variables stored in memory? When do they get deleted?

In this article, we’re going to do a deep dive into the internals of Python to understand how it handles memory management.

By the end of this article, you’ll:

Learn more about low-level computing, specifically as relates to memory
Understand how Python abstracts lower-level operations
Learn about Python’s internal memory management algorithms

Understanding Python’s internals will also give you better insight into some of Python’s behaviors. Hopefully, you’ll gain a new appreciation for Python as well. So much logic is happening behind the scenes to ensure your program works the way you expect.

Free Bonus: 5 Thoughts On Python Mastery, a free course for Python developers that shows you the roadmap and the mindset you'll need to take your Python skills to the next level.

Memory Is an Empty Book

You can begin by thinking of a computer’s memory as an empty book intended for short stories. There’s nothing written on the pages yet. Eventually, different authors will come along. Each author wants some space to write their story in.

Since they aren’t allowed to write over each other, they must be careful about which pages they write in. Before they begin writing, they consult the manager of the book. The manager then decides where in the book they’re allowed to write.

Since this book is around for a long time, many of the stories in it are no longer relevant. When no one reads or references the stories, they are removed to make room for new stories.

In essence, computer memory is like that empty book. In fact, it’s common to call fixed-length contiguous blocks of memory pages, so this analogy holds pretty well.

The authors are like different applications or processes that need to store data in memory. The manager, who decides where the authors can write in the book, plays the role of a memory manager of sorts. The person who removed the old stories to make room for new ones is a garbage collector.

Memory Management: From Hardware to Software

Memory management is the process by which applications read and write data. A memory manager determines where to put an application’s data. Since there’s a finite chunk of memory, like the pages in our book analogy, the manager has to find some free space and provide it to the application. This process of providing memory is generally called memory allocation.

On the flip side, when data is no longer needed, it can be deleted, or freed. But freed to where? Where did this “memory” come from?

Somewhere in your computer, there’s a physical device storing data when you’re running your Python programs. There are many layers of abstraction that the Python code goes through before the objects actually get to the hardware though.

One of the main layers above the hardware (such as RAM or a hard drive) is the operating system (OS). It carries out (or denies) requests to read and write memory.

Above the OS, there are applications, one of which is the default Python implementation (included in your OS or downloaded from python.org.) Memory management for your Python code is handled by the Python application. The algorithms and structures that the Python application uses for memory management is the focus of this article.

The Default Python Implementation

The default Python implementation, CPython, is actually written in the C programming language.

When I first heard this, it blew my mind. A language that’s written in another language?! Well, not really, but sort of.

The Python language is defined in a reference manual written in English. However, that manual isn’t all that useful by itself. You still need something to interpret written code based on the rules in the manual.

You also need something to actually execute interpreted code on a computer. The default Python implementation fulfills both of those requirements. It converts your Python code into instructions that it then runs on a virtual machine.

Note: Virtual machines are like physical computers, but they are implemented in software. They typically process basic instructions similar to Assembly instructions.

Python is an interpreted programming language. Your Python code actually gets compiled down to more computer-readable instructions called bytecode. These instructions get interpreted by a virtual machine when you run your code.

Have you ever seen a .pyc file or a __pycache__ folder? That’s the bytecode that gets interpreted by the virtual machine.

It’s important to note that there are implementations other than CPython. IronPython compiles down to run on Microsoft’s Common Language Runtime. Jython compiles down to Java bytecode to run on the Java Virtual Machine. Then there’s PyPy, but that deserves its own entire article, so I’ll just mention it in passing.

For the purposes of this article, I’ll focus on the memory management done by the default implementation of Python, CPython.

Disclaimer: While a lot of this information will carry through to new versions of Python, things may change in the future. Note that the referenced version for this article is the current latest version of Python, 3.7.

Okay, so CPython is written in C, and it interprets Python bytecode. What does this have to do with memory management? Well, the memory management algorithms and structures exist in the CPython code, in C. To understand the memory management of Python, you have to get a basic understanding of CPython itself.

CPython is written in C, which does not natively support object-oriented programming. Because of that, there are quite a bit of interesting designs in the CPython code.

You may have heard that everything in Python is an object, even types such as int and str. Well, it’s true on an implementation level in CPython. There is a struct called a PyObject, which every other object in CPython uses.

Note: A struct, or structure, in C is a custom data type that groups together different data types. To compare to object-oriented languages, it’s like a class with attributes and no methods.

The PyObject, the grand-daddy of all objects in Python, contains only two things:

ob_refcnt: reference count
ob_type: pointer to another type

The reference count is used for garbage collection. Then you have a pointer to the actual object type. That object type is just another struct that describes a Python object (such as a dict or int).

Each object has its own object-specific memory allocator that knows how to get the memory to store that object. Each object also has an object-specific memory deallocator that “frees” the memory once it’s no longer needed.

However, there’s an important factor in all this talk about allocating and freeing memory. Memory is a shared resource on the computer, and bad things can happen if two different processes try to write to the same location at the same time.

The Global Interpreter Lock (GIL)

The GIL is a solution to the common problem of dealing with shared resources, like memory in a computer. When two threads try to modify the same resource at the same time, they can step on each other’s toes. The end result can be a garbled mess where neither of the threads ends up with what they wanted.

Consider the book analogy again. Suppose that two authors stubbornly decide that it’s their turn to write. Not only that, but they both need to write on the same page of the book at the same time.

They each ignore the other’s attempt to craft a story and begin writing on the page. The end result is two stories on top of each other, which makes the whole page completely unreadable.

One solution to this problem is a single, global lock on the interpreter when a thread is interacting with the shared resource (the page in the book). In other words, only one author can write at a time.

Python’s GIL accomplishes this by locking the entire interpreter, meaning that it’s not possible for another thread to step on the current one. When CPython handles memory, it uses the GIL to ensure that it does so safely.

There are pros and cons to this approach, and the GIL is heavily debated in the Python community. To read more about the GIL, I suggest checking out What is the Python Global Interpreter Lock (GIL)?.

Garbage Collection

Let’s revisit the book analogy and assume that some of the stories in the book are getting very old. No one is reading or referencing those stories anymore. If no one is reading something or referencing it in their own work, you could get rid of it to make room for new writing.

That old, unreferenced writing could be compared to an object in Python whose reference count has dropped to 0. Remember that every object in Python has a reference count and a pointer to a type.

The reference count gets increased for a few different reasons. For example, the reference count will increase if you assign it to another variable:

numbers = [1, 2, 3]
# Reference count = 1
more_numbers = numbers
# Reference count = 2

It will also increase if you pass the object as an argument:

total = sum(numbers)

As a final example, the reference count will increase if you include the object in a list:

matrix = [numbers, numbers, numbers]

Python allows you to inspect the current reference count of an object with the sys module. You can use sys.getrefcount(numbers), but keep in mind that passing in the object to getrefcount() increases the reference count by 1.

In any case, if the object is still required to hang around in your code, its reference count is greater than 0. Once it drops to 0, the object has a specific deallocation function that is called which “frees” the memory so that other objects can use it.

But what does it mean to “free” the memory, and how do other objects use it? Let’s jump right into CPython’s memory management.

CPython’s Memory Management

We’re going to dive deep into CPython’s memory architecture and algorithms, so buckle up.

As mentioned before, there are layers of abstraction from the physical hardware to CPython. The operating system (OS) abstracts the physical memory and creates a virtual memory layer that applications (including Python) can access.

An OS-specific virtual memory manager carves out a chunk of memory for the Python process. The darker gray boxes in the image below are now owned by the Python process.

Python uses a portion of the memory for internal use and non-object memory. The other portion is dedicated to object storage (your int, dict, and the like). Note that this was somewhat simplified. If you want the full picture, you can check out the CPython source code, where all this memory management happens.

CPython has an object allocator that is responsible for allocating memory within the object memory area. This object allocator is where most of the magic happens. It gets called every time a new object needs space allocated or deleted.

Typically, the adding and removing of data for Python objects like list and int doesn’t involve too much data at a time. So the design of the allocator is tuned to work well with small amounts of data at a time. It also tries not to allocate memory until it’s absolutely required.

The comments in the source code describe the allocator as “a fast, special-purpose memory allocator for small blocks, to be used on top of a general-purpose malloc.” In this case, malloc is C’s library function for memory allocation.

Now we’ll look at CPython’s memory allocation strategy. First, we’ll talk about the 3 main pieces and how they relate to each other.

Arenas are the largest chunks of memory and are aligned on a page boundary in memory. A page boundary is the edge of a fixed-length contiguous chunk of memory that the OS uses. Python assumes the system’s page size is 256 kilobytes.

Within the arenas are pools, which are one virtual memory page (4 kilobytes). These are like the pages in our book analogy. These pools are fragmented into smaller blocks of memory.

All the blocks in a given pool are of the same “size class.” A size class defines a specific block size, given some amount of requested data. The chart below is taken directly from the source code comments:

Request in bytes	Size of allocated block	Size class idx
1-8	8	0
9-16	16	1
17-24	24	2
25-32	32	3
33-40	40	4
41-48	48	5
49-56	56	6
57-64	64	7
65-72	72	8
…	…	…
497-504	504	62
505-512	512	63

For example, if 42 bytes are requested, the data would be placed into a size 48-byte block.

Pools

Pools are composed of blocks from a single size class. Each pool maintains a double-linked list to other pools of the same size class. In that way, the algorithm can easily find available space for a given block size, even across different pools.

A usedpools list tracks all the pools that have some space available for data for each size class. When a given block size is requested, the algorithm checks this usedpools list for the list of pools for that block size.

Pools themselves must be in one of 3 states: used, full, or empty. A used pool has available blocks for data to be stored. A full pool’s blocks are all allocated and contain data. An empty pool has no data stored and can be assigned any size class for blocks when needed.

A freepools list keeps track of all the pools in the empty state. But when do empty pools get used?

Assume your code needs an 8-byte chunk of memory. If there are no pools in usedpools of the 8-byte size class, a fresh empty pool is initialized to store 8-byte blocks. This new pool then gets added to the usedpools list so it can be used for future requests.

Say a full pool frees some of its blocks because the memory is no longer needed. That pool would get added back to the usedpools list for its size class.

You can see now how pools can move between these states (and even memory size classes) freely with this algorithm.

Blocks

As seen in the diagram above, pools contain a pointer to their “free” blocks of memory. There’s a slight nuance to the way this works. This allocator “strives at all levels (arena, pool, and block) never to touch a piece of memory until it’s actually needed,” according to the comments in the source code.

That means that a pool can have blocks in 3 states. These states can be defined as follows:

untouched: a portion of memory that has not been allocated
free: a portion of memory that was allocated but later made “free” by CPython and that no longer contains relevant data
allocated: a portion of memory that actually contains relevant data

The freeblock pointer points to a singly linked list of free blocks of memory. In other words, a list of available places to put data. If more than the available free blocks are needed, the allocator will get some untouched blocks in the pool.

As the memory manager makes blocks “free,” those now free blocks get added to the front of the freeblock list. The actual list may not be contiguous blocks of memory, like the first nice diagram. It may look something like the diagram below:

Arenas

Arenas contain pools. Those pools can be used, full, or empty. Arenas themselves don’t have as explicit states as pools do though.

Arenas are instead organized into a doubly linked list called usable_arenas. The list is sorted by the number of free pools available. The fewer free pools, the closer the arena is to the front of the list.

This means that the arena that is the most full of data will be selected to place new data into. But why not the opposite? Why not place data where there’s the most available space?

This brings us to the idea of truly freeing memory. You’ll notice that I’ve been saying “free” in quotes quite a bit. The reason is that when a block is deemed “free”, that memory is not actually freed back to the operating system. The Python process keeps it allocated and will use it later for new data. Truly freeing memory returns it to the operating system to use.

Arenas are the only things that can truly be freed. So, it stands to reason that those arenas that are closer to being empty should be allowed to become empty. That way, that chunk of memory can be truly freed, reducing the overall memory footprint of your Python program.

Conclusion

Memory management is an integral part of working with computers. Python handles nearly all of it behind the scenes, for better or for worse.

In this article, you learned:

What memory management is and why it’s important
How the default Python implementation, CPython, is written in the C programming language
How the data structures and algorithms work together in CPython’s memory management to handle your data

Python abstracts away a lot of the gritty details of working with computers. This gives you the power to work on a higher level to develop your code without the headache of worrying about how and where all those bytes are getting stored.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Interactive Data Visualization in Python With Bokeh

2018-11-19T14:00:00+00:00

Bokeh prides itself on being a library for interactive data visualization.

Unlike popular counterparts in the Python visualization space, like Matplotlib and Seaborn, Bokeh renders its graphics using HTML and JavaScript. This makes it a great candidate for building web-based dashboards and applications. However, it’s an equally powerful tool for exploring and understanding your data or creating beautiful custom charts for a project or report.

Using a number of examples on a real-world dataset, the goal of this tutorial is to get you up and running with Bokeh.

You’ll learn how to:

Transform your data into visualizations, using Bokeh
Customize and organize your visualizations
Add interactivity to your visualizations

So let’s jump in.

From Data to Visualization

Building a visualization with Bokeh involves the following steps:

Prepare the data
Determine where the visualization will be rendered
Set up the figure(s)
Connect to and draw your data
Organize the layout
Preview and save your beautiful data creation

Let’s explore each step in more detail.

Prepare the Data

Any good data visualization starts with—you guessed it—data. If you need a quick refresher on handling data in Python, definitely check out the growing number of excellent Real Python tutorials on the subject.

This step commonly involves data handling libraries like Pandas and Numpy and is all about taking the required steps to transform it into a form that is best suited for your intended visualization.

Determine Where the Visualization Will Be Rendered

At this step, you’ll determine how you want to generate and ultimately view your visualization. In this tutorial, you’ll learn about two common options that Bokeh provides: generating a static HTML file and rendering your visualization inline in a Jupyter Notebook.

Set up the Figure(s)

From here, you’ll assemble your figure, preparing the canvas for your visualization. In this step, you can customize everything from the titles to the tick marks. You can also set up a suite of tools that can enable various user interactions with your visualization.

Connect to and Draw Your Data

Next, you’ll use Bokeh’s multitude of renderers to give shape to your data. Here, you have the flexibility to draw your data from scratch using the many available marker and shape options, all of which are easily customizable. This functionality gives you incredible creative freedom in representing your data.

Additionally, Bokeh has some built-in functionality for building things like stacked bar charts and plenty of examples for creating more advanced visualizations like network graphs and maps.

Organize the Layout

If you need more than one figure to express your data, Bokeh’s got you covered. Not only does Bokeh offer the standard grid-like layout options, but it also allows you to easily organize your visualizations into a tabbed layout in just a few lines of code.

In addition, your plots can be quickly linked together, so a selection on one will be reflected on any combination of the others.

Preview and Save Your Beautiful Data Creation

Finally, it’s time to see what you created.

Whether you’re viewing your visualization in a browser or notebook, you’ll be able to explore your visualization, examine your customizations, and play with any interactions that were added.

If you like what you see, you can save your visualization to an image file. Otherwise, you can revisit the steps above as needed to bring your data vision to reality.

That’s it! Those six steps are the building blocks for a tidy, flexible template that can be used to take your data from the table to the big screen:

"""Bokeh Visualization Template

This template is a general outline for turning your data into a 
visualization using Bokeh.
"""
# Data handling
import pandas as pd
import numpy as np

# Bokeh libraries
from bokeh.io import output_file, output_notebook
from bokeh.plotting import figure, show
from bokeh.models import ColumnDataSource
from bokeh.layouts import row, column, gridplot
from bokeh.models.widgets import Tabs, Panel

# Prepare the data

# Determine where the visualization will be rendered
output_file('filename.html')  # Render to static HTML, or 
output_notebook()  # Render inline in a Jupyter Notebook

# Set up the figure(s)
fig = figure()  # Instantiate a figure() object

# Connect to and draw the data

# Organize the layout

# Preview and save 
show(fig)  # See what I made, and save if I like it

Some common code snippets that are found in each step are previewed above, and you’ll see how to fill out the rest as you move through the rest of the tutorial!

Generating Your First Figure

There are multiple ways to output your visualization in Bokeh. In this tutorial, you’ll see these two options:

output_file('filename.html') will write the visualization to a static HTML file.
output_notebook() will render your visualization directly in a Jupyter Notebook.

It’s important to note that neither function will actually show you the visualization. That doesn’t happen until show() is called. However, they will ensure that, when show() is called, the visualization appears where you intend it to.

By calling both output_file() and output_notebook() in the same execution, the visualization will be rendered both to a static HTML file and inline in the notebook. However, if for whatever reason you run multiple output_file() commands in the same execution, only the last one will be used for rendering.

This is a great opportunity to give you your first glimpse at a default Bokeh figure() using output_file():

# Bokeh Libraries
from bokeh.io import output_file
from bokeh.plotting import figure, show

# The figure will be rendered in a static HTML file called output_file_test.html
output_file('output_file_test.html', 
            title='Empty Bokeh Figure')

# Set up a generic figure() object
fig = figure()

# See what it looks like
show(fig)

As you can see, a new browser window opened with a tab called Empty Bokeh Figure and an empty figure. Not shown is the file generated with the name output_file_test.html in your current working directory.

If you were to run the same code snippet with output_notebook() in place of output_file(), assuming you have a Jupyter Notebook fired up and ready to go, you will get the following:

# Bokeh Libraries
from bokeh.io import output_notebook
from bokeh.plotting import figure, show

# The figure will be right in my Jupyter Notebook
output_notebook()

# Set up a generic figure() object
fig = figure()

# See what it looks like
show(fig)

As you can see, the result is the same, just rendered in a different location.

More information about both output_file() and output_notebook() can be found in the Bokeh official docs.

Note: Sometimes, when rendering multiple visualizations sequentially, you’ll see that past renders are not being cleared with each execution. If you experience this, import and run the following between executions:

# Import reset_output (only needed once) 
from bokeh.plotting import reset_output

# Use reset_output() between subsequent show() calls, as needed
reset_output()

Before moving on, you may have noticed that the default Bokeh figure comes pre-loaded with a toolbar. This is an important sneak preview into the interactive elements of Bokeh that come right out of the box. You’ll find out more about the toolbar and how to configure it in the Adding Interaction section at the end of this tutorial.

Getting Your Figure Ready for Data

Now that you know how to create and view a generic Bokeh figure either in a browser or Jupyter Notebook, it’s time to learn more about how to configure the figure() object.

The figure() object is not only the foundation of your data visualization but also the object that unlocks all of Bokeh’s available tools for visualizing data. The Bokeh figure is a subclass of the Bokeh Plot object, which provides many of the parameters that make it possible to configure the aesthetic elements of your figure.

To show you just a glimpse into the customization options available, let’s create the ugliest figure ever:

# Bokeh Libraries
from bokeh.io import output_notebook
from bokeh.plotting import figure, show

# The figure will be rendered inline in my Jupyter Notebook
output_notebook()

# Example figure
fig = figure(background_fill_color='gray',
             background_fill_alpha=0.5,
             border_fill_color='blue',
             border_fill_alpha=0.25,
             plot_height=300,
             plot_width=500,
             h_symmetry=True,
             x_axis_label='X Label',
             x_axis_type='datetime',
             x_axis_location='above',
             x_range=('2018-01-01', '2018-06-30'),
             y_axis_label='Y Label',
             y_axis_type='linear',
             y_axis_location='left',
             y_range=(0, 100),
             title='Example Figure',
             title_location='right',
             toolbar_location='below',
             tools='save')

# See what it looks like
show(fig)

Once the figure() object is instantiated, you can still configure it after the fact. Let’s say you want to get rid of the gridlines:

# Remove the gridlines from the figure() object
fig.grid.grid_line_color = None

# See what it looks like 
show(fig)

The gridline properties are accessible via the figure’s grid attribute. In this case, setting grid_line_color to None effectively removes the gridlines altogether. More details about figure attributes can be found below the fold in the Plot class documentation.

Note: If you’re working in a notebook or IDE with auto-complete functionality, this feature can definitely be your friend! With so many customizable elements, it can be very helpful in discovering the available options:

Otherwise, doing a quick web search, with the keyword bokeh and what you are trying to do, will generally point you in the right direction.

There is tons more I could touch on here, but don’t feel like you’re missing out. I’ll make sure to introduce different figure tweaks as the tutorial progresses. Here are some other helpful links on the topic:

The Bokeh Plot Class is the superclass of the figure() object, from which figures inherit a lot of their attributes.
The Figure Class documentation is a good place to find more detail about the arguments of the figure() object.

Here are a few specific customization options worth checking out:

Text Properties covers all the attributes related to changing font styles, sizes, colors, and so forth.
TickFormatters are built-in objects specifically for formatting your axes using Python-like string formatting syntax.

Sometimes, it isn’t clear how your figure needs to be customized until it actually has some data visualized in it, so next you’ll learn how to make that happen.

Drawing Data With Glyphs

An empty figure isn’t all that exciting, so let’s look at glyphs: the building blocks of Bokeh visualizations. A glyph is a vectorized graphical shape or marker that is used to represent your data, like a circle or square. More examples can be found in the Bokeh gallery. After you create your figure, you are given access to a bevy of configurable glyph methods.

Let’s start with a very basic example, drawing some points on an x-y coordinate grid:

# Bokeh Libraries
from bokeh.io import output_file
from bokeh.plotting import figure, show

# My x-y coordinate data
x = [1, 2, 1]
y = [1, 1, 2]

# Output the visualization directly in the notebook
output_file('first_glyphs.html', title='First Glyphs')

# Create a figure with no toolbar and axis ranges of [0,3]
fig = figure(title='My Coordinates',
             plot_height=300, plot_width=300,
             x_range=(0, 3), y_range=(0, 3),
             toolbar_location=None)

# Draw the coordinates as circles
fig.circle(x=x, y=y,
           color='green', size=10, alpha=0.5)

# Show plot
show(fig)

Once your figure is instantiated, you can see how it can be used to draw the x-y coordinate data using customized circle glyphs.

Here are a few categories of glyphs:

Marker includes shapes like circles, diamonds, squares, and triangles and is effective for creating visualizations like scatter and bubble charts.
Line covers things like single, step, and multi-line shapes that can be used to build line charts.
Bar/Rectangle shapes can be used to create traditional or stacked bar (hbar) and column (vbar) charts as well as waterfall or gantt charts.

Information about the glyphs above, as well as others, can be found in Bokeh’s Reference Guide.

These glyphs can be combined as needed to fit your visualization needs. Let’s say I want to create a visualization that shows how many words I wrote per day to make this tutorial, with an overlaid trend line of the cumulative word count:

import numpy as np

# Bokeh libraries
from bokeh.io import output_notebook
from bokeh.plotting import figure, show

# My word count data
day_num = np.linspace(1, 10, 10)
daily_words = [450, 628, 488, 210, 287, 791, 508, 639, 397, 943]
cumulative_words = np.cumsum(daily_words)

# Output the visualization directly in the notebook
output_notebook()

# Create a figure with a datetime type x-axis
fig = figure(title='My Tutorial Progress',
             plot_height=400, plot_width=700,
             x_axis_label='Day Number', y_axis_label='Words Written',
             x_minor_ticks=2, y_range=(0, 6000),
             toolbar_location=None)

# The daily words will be represented as vertical bars (columns)
fig.vbar(x=day_num, bottom=0, top=daily_words, 
         color='blue', width=0.75, 
         legend='Daily')

# The cumulative sum will be a trend line
fig.line(x=day_num, y=cumulative_words, 
         color='gray', line_width=1,
         legend='Cumulative')

# Put the legend in the upper left corner
fig.legend.location = 'top_left'

# Let's check it out
show(fig)

To combine the columns and lines on the figure, they are simply created using the same figure() object.

Additionally, you can see above how seamlessly a legend can be created by setting the legend property for each glyph. The legend was then moved to the upper left corner of the plot by assigning 'top_left' to fig.legend.location.

You can check out much more info about styling legends. Teaser: they will show up again later in the tutorial when we start digging into interactive elements of the visualization.

A Quick Aside About Data

Anytime you are exploring a new visualization library, it’s a good idea to start with some data in a domain you are familiar with. The beauty of Bokeh is that nearly any idea you have should be possible. It’s just a matter of how you want to leverage the available tools to do so.

The remaining examples will use publicly available data from Kaggle, which has information about the National Basketball Association’s (NBA) 2017-18 season, specifically:

2017-18_playerBoxScore.csv: game-by-game snapshots of player statistics
2017-18_teamBoxScore.csv: game-by-game snapshots of team statistics
2017-18_standings.csv: daily team standings and rankings

This data has nothing to do with what I do for work, but I love basketball and enjoy thinking about ways to visualize the ever-growing amount of data associated with it.

If you don’t have data to play with from school or work, think about something you’re interested in and try to find some data related to that. It will go a long way in making both the learning and the creative process faster and more enjoyable!

To follow along with the examples in the tutorial, you can download the datasets from the links above and read them into a Pandas DataFrame using the following commands:

import pandas as pd

# Read the csv files
player_stats = pd.read_csv('2017-18_playerBoxScore.csv', parse_dates=['gmDate'])
team_stats = pd.read_csv('2017-18_teamBoxScore.csv', parse_dates=['gmDate'])
standings = pd.read_csv('2017-18_standings.csv', parse_dates=['stDate'])

This code snippet reads the data from the three CSV files and automatically interprets the date columns as datetime objects.

It’s now time to get your hands on some real data.

Using the `ColumnDataSource` Object

The examples above used Python lists and Numpy arrays to represent the data, and Bokeh is well equipped to handle these datatypes. However, when it comes to data in Python, you are most likely going to come across Python dictionaries and Pandas DataFrames, especially if you’re reading in data from a file or external data source.

Bokeh is well equipped to work with these more complex data structures and even has built-in functionality to handle them, namely the ColumnDataSource.

You may be asking yourself, “Why use a ColumnDataSource when Bokeh can interface with other data types directly?”

For one, whether you reference a list, array, dictionary, or DataFrame directly, Bokeh is going to turn it into a ColumnDataSource behind the scenes anyway. More importantly, the ColumnDataSource makes it much easier to implement Bokeh’s interactive affordances.

The ColumnDataSource is foundational in passing the data to the glyphs you are using to visualize. Its primary functionality is to map names to the columns of your data. This makes it easier for you to reference elements of your data when building your visualization. It also makes it easier for Bokeh to do the same when building your visualization.

The ColumnDataSource can interpret three types of data objects:

Python dict: The keys are names associated with the respective value sequences (lists, arrays, and so forth).
Pandas DataFrame: The columns of the DataFrame become the reference names for the ColumnDataSource.
Pandas groupby: The columns of the ColumnDataSource reference the columns as seen by calling groupby.describe().

Let’s start by visualizing the race for first place in the NBA’s Western Conference in 2017-18 between the defending champion Golden State Warriors and the challenger Houston Rockets. The daily win-loss records of these two teams is stored in a DataFrame named west_top_2:

>>>

>>> west_top_2 = (standings[(standings['teamAbbr'] == 'HOU') | (standings['teamAbbr'] == 'GS')]
...               .loc[:, ['stDate', 'teamAbbr', 'gameWon']]
...               .sort_values(['teamAbbr','stDate']))
>>> west_top_2.head()
        stDate teamAbbr  gameWon
9   2017-10-17       GS        0
39  2017-10-18       GS        0
69  2017-10-19       GS        0
99  2017-10-20       GS        1
129 2017-10-21       GS        1

From here, you can load this DataFrame into two ColumnDataSource objects and visualize the race:

# Bokeh libraries
from bokeh.plotting import figure, show
from bokeh.io import output_file
from bokeh.models import ColumnDataSource

# Output to file
output_file('west-top-2-standings-race.html', 
            title='Western Conference Top 2 Teams Wins Race')

# Isolate the data for the Rockets and Warriors
rockets_data = west_top_2[west_top_2['teamAbbr'] == 'HOU']
warriors_data = west_top_2[west_top_2['teamAbbr'] == 'GS']

# Create a ColumnDataSource object for each team
rockets_cds = ColumnDataSource(rockets_data)
warriors_cds = ColumnDataSource(warriors_data)

# Create and configure the figure
fig = figure(x_axis_type='datetime',
             plot_height=300, plot_width=600,
             title='Western Conference Top 2 Teams Wins Race, 2017-18',
             x_axis_label='Date', y_axis_label='Wins',
             toolbar_location=None)

# Render the race as step lines
fig.step('stDate', 'gameWon', 
         color='#CE1141', legend='Rockets', 
         source=rockets_cds)
fig.step('stDate', 'gameWon', 
         color='#006BB6', legend='Warriors', 
         source=warriors_cds)

# Move the legend to the upper left corner
fig.legend.location = 'top_left'

# Show the plot
show(fig)

Notice how the respective ColumnDataSource objects are referenced when creating the two lines. You simply pass the original column names as input parameters and specify which ColumnDataSource to use via the source property.

The visualization shows the tight race throughout the season, with the Warriors building a pretty big cushion around the middle of the season. However, a bit of a late-season slide allowed the Rockets to catch up and ultimately surpass the defending champs to finish the season as the Western Conference number-one seed.

Note: In Bokeh, you can specify colors either by name, hex value, or RGB color code.

For the visualization above, a color is being specified for the respective lines representing the two teams. Instead of using CSS color names like 'red' for the Rockets and 'blue' for the Warriors, you might have wanted to add a nice visual touch by using the official team colors in the form of hex color codes. Alternatively, you could have used tuples representing RGB color codes: (206, 17, 65) for the Rockets, (0, 107, 182) for the Warriors.

Bokeh provides a helpful list of CSS color names categorized by their general hue. Also, htmlcolorcodes.com is a great site for finding CSS, hex, and RGB color codes.

ColumnDataSource objects can do more than just serve as an easy way to reference DataFrame columns. The ColumnDataSource object has three built-in filters that can be used to create views on your data using a CDSView object:

GroupFilter selects rows from a ColumnDataSource based on a categorical reference value
IndexFilter filters the ColumnDataSource via a list of integer indices
BooleanFilter allows you to use a list of boolean values, with True rows being selected

In the previous example, two ColumnDataSource objects were created, one each from a subset of the west_top_2 DataFrame. The next example will recreate the same output from one ColumnDataSource based on all of west_top_2 using a GroupFilter that creates a view on the data:

# Bokeh libraries
from bokeh.plotting import figure, show
from bokeh.io import output_file
from bokeh.models import ColumnDataSource, CDSView, GroupFilter

# Output to file
output_file('west-top-2-standings-race.html', 
            title='Western Conference Top 2 Teams Wins Race')

# Create a ColumnDataSource
west_cds = ColumnDataSource(west_top_2)

# Create views for each team
rockets_view = CDSView(source=west_cds,
                       filters=[GroupFilter(column_name='teamAbbr', group='HOU')])
warriors_view = CDSView(source=west_cds,
                        filters=[GroupFilter(column_name='teamAbbr', group='GS')])

# Create and configure the figure
west_fig = figure(x_axis_type='datetime',
                  plot_height=300, plot_width=600,
                  title='Western Conference Top 2 Teams Wins Race, 2017-18',
                  x_axis_label='Date', y_axis_label='Wins',
                  toolbar_location=None)

# Render the race as step lines
west_fig.step('stDate', 'gameWon',
              source=west_cds, view=rockets_view,
              color='#CE1141', legend='Rockets')
west_fig.step('stDate', 'gameWon',
              source=west_cds, view=warriors_view,
              color='#006BB6', legend='Warriors')

# Move the legend to the upper left corner
west_fig.legend.location = 'top_left'

# Show the plot
show(west_fig)

Notice how the GroupFilter is passed to CDSView in a list. This allows you to combine multiple filters together to isolate the data you need from the ColumnDataSource as needed.

For information about integrating data sources, check out the Bokeh user guide’s post on the ColumnDataSource and other source objects available.

The Western Conference ended up being an exciting race, but say you want to see if the Eastern Conference was just as tight. Not only that, but you’d like to view them in a single visualization. This is a perfect segue to the next topic: layouts.

Organizing Multiple Visualizations With Layouts

The Eastern Conference standings came down to two rivals in the Atlantic Division: the Boston Celtics and the Toronto Raptors. Before replicating the steps used to create west_top_2, let’s try to put the ColumnDataSource to the test one more time using what you learned above.

In this example, you’ll see how to feed an entire DataFrame into a ColumnDataSource and create views to isolate the relevant data:

# Bokeh libraries
from bokeh.plotting import figure, show
from bokeh.io import output_file
from bokeh.models import ColumnDataSource, CDSView, GroupFilter

# Output to file
output_file('east-top-2-standings-race.html', 
            title='Eastern Conference Top 2 Teams Wins Race')

# Create a ColumnDataSource
standings_cds = ColumnDataSource(standings)

# Create views for each team
celtics_view = CDSView(source=standings_cds,
                      filters=[GroupFilter(column_name='teamAbbr', 
                                           group='BOS')])
raptors_view = CDSView(source=standings_cds,
                      filters=[GroupFilter(column_name='teamAbbr', 
                                           group='TOR')])

# Create and configure the figure
east_fig = figure(x_axis_type='datetime',
           plot_height=300, plot_width=600,
           title='Eastern Conference Top 2 Teams Wins Race, 2017-18',
           x_axis_label='Date', y_axis_label='Wins',
           toolbar_location=None)

# Render the race as step lines
east_fig.step('stDate', 'gameWon', 
              color='#007A33', legend='Celtics',
              source=standings_cds, view=celtics_view)
east_fig.step('stDate', 'gameWon', 
              color='#CE1141', legend='Raptors',
              source=standings_cds, view=raptors_view)

# Move the legend to the upper left corner
east_fig.legend.location = 'top_left'

# Show the plot
show(east_fig)

The ColumnDataSource was able to isolate the relevant data within a 5,040-by-39 DataFrame without breaking a sweat, saving a few lines of Pandas code in the process.

Looking at the visualization, you can see that the Eastern Conference race was no slouch. After the Celtics roared out of the gate, the Raptors clawed all the way back to overtake their division rival and finish the regular season with five more wins.

With our two visualizations ready, it’s time to put them together.

Similar to the functionality of Matplotlib’s subplot, Bokeh offers the column, row, and gridplot functions in its bokeh.layouts module. These functions can more generally be classified as layouts.

The usage is very straightforward. If you want to put two visualizations in a vertical configuration, you can do so with the following:

# Bokeh library
from bokeh.plotting import figure, show
from bokeh.io import output_file
from bokeh.layouts import column

# Output to file
output_file('east-west-top-2-standings-race.html', 
            title='Conference Top 2 Teams Wins Race')

# Plot the two visualizations in a vertical configuration
show(column(west_fig, east_fig))

I’ll save you the two lines of code, but rest assured that swapping column for row in the snippet above will similarly configure the two plots in a horizontal configuration.

Note: If you’re trying out the code snippets as you go through the tutorial, I want to take a quick detour to address an error you may see accessing west_fig and east_fig in the following examples. In doing so, you may receive an error like this:

WARNING:bokeh.core.validation.check:W-1004 (BOTH_CHILD_AND_ROOT): Models should not be a document root...

This is one of many errors that are part of Bokeh’s validation module, where w-1004 in particular is warning about the re-use of west_fig and east_fig in a new layout.

To avoid this error as you test the examples, preface the code snippet illustrating each layout with the following:

# Bokeh libraries
from bokeh.plotting import figure, show
from bokeh.models import ColumnDataSource, CDSView, GroupFilter

# Create a ColumnDataSource
standings_cds = ColumnDataSource(standings)

# Create the views for each team
celtics_view = CDSView(source=standings_cds,
                      filters=[GroupFilter(column_name='teamAbbr', 
                                           group='BOS')])

raptors_view = CDSView(source=standings_cds,
                      filters=[GroupFilter(column_name='teamAbbr', 
                                           group='TOR')])

rockets_view = CDSView(source=standings_cds,
                      filters=[GroupFilter(column_name='teamAbbr', 
                                           group='HOU')])
warriors_view = CDSView(source=standings_cds,
                      filters=[GroupFilter(column_name='teamAbbr', 
                                           group='GS')])

# Create and configure the figure
east_fig = figure(x_axis_type='datetime',
                  plot_height=300,
                  x_axis_label='Date',
                  y_axis_label='Wins',
                  toolbar_location=None)

west_fig = figure(x_axis_type='datetime',
                  plot_height=300,
                  x_axis_label='Date',
                  y_axis_label='Wins',
                  toolbar_location=None)

# Configure the figures for each conference
east_fig.step('stDate', 'gameWon', 
              color='#007A33', legend='Celtics',
              source=standings_cds, view=celtics_view)
east_fig.step('stDate', 'gameWon', 
              color='#CE1141', legend='Raptors',
              source=standings_cds, view=raptors_view)

west_fig.step('stDate', 'gameWon', color='#CE1141', legend='Rockets',
              source=standings_cds, view=rockets_view)
west_fig.step('stDate', 'gameWon', color='#006BB6', legend='Warriors',
              source=standings_cds, view=warriors_view)

# Move the legend to the upper left corner
east_fig.legend.location = 'top_left'
west_fig.legend.location = 'top_left'

# Layout code snippet goes here!

Doing so will renew the relevant components to render the visualization, ensuring that no warning is needed.

Instead of using column or row, you may want to use a gridplot instead.

One key difference of gridplot is that it will automatically consolidate the toolbar across all of its children figures. The two visualizations above do not have a toolbar, but if they did, then each figure would have its own when using column or row. With that, it also has its own toolbar_location property, seen below set to 'right'.

Syntactically, you’ll also notice below that gridplot differs in that, instead of being passed a tuple as input, it requires a list of lists, where each sub-list represents a row in the grid:

# Bokeh libraries
from bokeh.io import output_file
from bokeh.layouts import gridplot

# Output to file
output_file('east-west-top-2-gridplot.html', 
            title='Conference Top 2 Teams Wins Race')

# Reduce the width of both figures
east_fig.plot_width = west_fig.plot_width = 300

# Edit the titles
east_fig.title.text = 'Eastern Conference'
west_fig.title.text = 'Western Conference'

# Configure the gridplot
east_west_gridplot = gridplot([[west_fig, east_fig]], 
                              toolbar_location='right')

# Plot the two visualizations in a horizontal configuration
show(east_west_gridplot)

Lastly, gridplot allows the passing of None values, which are interpreted as blank subplots. Therefore, if you wanted to leave a placeholder for two additional plots, then you could do something like this:

# Bokeh libraries
from bokeh.io import output_file
from bokeh.layouts import gridplot

# Output to file
output_file('east-west-top-2-gridplot.html', 
            title='Conference Top 2 Teams Wins Race')

# Reduce the width of both figures
east_fig.plot_width = west_fig.plot_width = 300

# Edit the titles
east_fig.title.text = 'Eastern Conference'
west_fig.title.text = 'Western Conference'

# Plot the two visualizations with placeholders
east_west_gridplot = gridplot([[west_fig, None], [None, east_fig]], 
                              toolbar_location='right')

# Plot the two visualizations in a horizontal configuration
show(east_west_gridplot)

If you’d rather toggle between both visualizations at their full size without having to squash them down to fit next to or on top of each other, a good option is a tabbed layout.

A tabbed layout consists of two Bokeh widget functions: Tab() and Panel() from the bokeh.models.widgets sub-module. Like using gridplot(), making a tabbed layout is pretty straightforward:

# Bokeh Library
from bokeh.io import output_file
from bokeh.models.widgets import Tabs, Panel

# Output to file
output_file('east-west-top-2-tabbed_layout.html', 
            title='Conference Top 2 Teams Wins Race')

# Increase the plot widths
east_fig.plot_width = west_fig.plot_width = 800

# Create two panels, one for each conference
east_panel = Panel(child=east_fig, title='Eastern Conference')
west_panel = Panel(child=west_fig, title='Western Conference')

# Assign the panels to Tabs
tabs = Tabs(tabs=[west_panel, east_panel])

# Show the tabbed layout
show(tabs)

The first step is to create a Panel() for each tab. That may sound a little confusing, but think of the Tabs() function as the mechanism that organizes the individual tabs created with Panel().

Each Panel() takes as input a child, which can either be a single figure() or a layout. (Remember that a layout is a general name for a column, row, or gridplot.) Once your panels are assembled, they can be passed as input to Tabs() in a list.

Now that you understand how to access, draw, and organize your data, it’s time to move on to the real magic of Bokeh: interaction! As always, check out Bokeh’s User Guide for more information on layouts.

Adding Interaction

The feature that sets Bokeh apart is its ability to easily implement interactivity in your visualization. Bokeh even goes as far as describing itself as an interactive visualization library:

Bokeh is an interactive visualization library that targets modern web browsers for presentation. (Source)

In this section, we’ll touch on five ways that you can add interactivity:

Configuring the toolbar
Selecting data points
Adding hover actions
Linking axes and selections
Highlighting data using the legend

Implementing these interactive elements open up possibilities for exploring your data that static visualizations just can’t do by themselves.

As you saw all the way back in Generating Your First Figure, the default Bokeh figure() comes with a toolbar right out of the box. The default toolbar comes with the following tools (from left to right):

Pan
Box Zoom
Wheel Zoom
Save
Reset
A link to Bokeh’s user guide for Configuring Plot Tools
A link to the Bokeh homepage

The toolbar can be removed by passing toolbar_location=None when instantiating a figure() object, or relocated by passing any of 'above', 'below', 'left', or 'right'.

Additionally, the toolbar can be configured to include any combination of tools you desire. Bokeh offers 18 specific tools across five categories:

Pan/Drag: box_select, box_zoom, lasso_select, pan, xpan, ypan, resize_select
Click/Tap: poly_select, tap
Scroll/Pinch: wheel_zoom, xwheel_zoom, ywheel_zoom
Actions: undo, redo, reset, save
Inspectors: crosshair, hover

To geek out on tools , make sure to visit Specifying Tools. Otherwise, they’ll be illustrated in covering the various interactions covered herein.

Selecting Data Points

Implementing selection behavior is as easy as adding a few specific keywords when declaring your glyphs.

The next example will create a scatter plot that relates a player’s total number of three-point shot attempts to the percentage made (for players with at least 100 three-point shot attempts).

The data can be aggregated from the player_stats DataFrame:

# Find players who took at least 1 three-point shot during the season
three_takers = player_stats[player_stats['play3PA'] > 0]

# Clean up the player names, placing them in a single column
three_takers['name'] = [f'{p["playFNm"]} {p["playLNm"]}' 
                        for _, p in three_takers.iterrows()]

# Aggregate the total three-point attempts and makes for each player
three_takers = (three_takers.groupby('name')
                            .sum()
                            .loc[:,['play3PA', 'play3PM']]
                            .sort_values('play3PA', ascending=False))

# Filter out anyone who didn't take at least 100 three-point shots
three_takers = three_takers[three_takers['play3PA'] >= 100].reset_index()

# Add a column with a calculated three-point percentage (made/attempted)
three_takers['pct3PM'] = three_takers['play3PM'] / three_takers['play3PA']

Here’s a sample of the resulting DataFrame:

>>>

>>> three_takers.sample(5)
                   name  play3PA  play3PM    pct3PM
229        Corey Brewer      110       31  0.281818
78           Marc Gasol      320      109  0.340625
126      Raymond Felton      230       81  0.352174
127  Kristaps Porziņģis      229       90  0.393013
66      Josh Richardson      336      127  0.377976

Let’s say you want to select a groups of players in the distribution, and in doing so mute the color of the glyphs representing the non-selected players:

# Bokeh Libraries
from bokeh.plotting import figure, show
from bokeh.io import output_file
from bokeh.models import ColumnDataSource, NumeralTickFormatter

# Output to file
output_file('three-point-att-vs-pct.html',
            title='Three-Point Attempts vs. Percentage')

# Store the data in a ColumnDataSource
three_takers_cds = ColumnDataSource(three_takers)

# Specify the selection tools to be made available
select_tools = ['box_select', 'lasso_select', 'poly_select', 'tap', 'reset']

# Create the figure
fig = figure(plot_height=400,
             plot_width=600,
             x_axis_label='Three-Point Shots Attempted',
             y_axis_label='Percentage Made',
             title='3PT Shots Attempted vs. Percentage Made (min. 100 3PA), 2017-18',
             toolbar_location='below',
             tools=select_tools)

# Format the y-axis tick labels as percentages
fig.yaxis[0].formatter = NumeralTickFormatter(format='00.0%')

# Add square representing each player
fig.square(x='play3PA',
           y='pct3PM',
           source=three_takers_cds,
           color='royalblue',
           selection_color='deepskyblue',
           nonselection_color='lightgray',
           nonselection_alpha=0.3)

# Visualize
show(fig)

First, specify the selection tools you want to make available. In the example above, 'box_select', 'lasso_select', 'poly_select', and 'tap' (plus a reset button) were specified in a list called select_tools. When the figure is instantiated, the toolbar is positioned 'below' the plot, and the list is passed to tools to make the tools selected above available.

Each player is initially represented by a royal blue square glyph, but the following configurations are set for when a player or group of players is selected:

Turn the selected player(s) to deepskyblue
Change all non-selected players’ glyphs to a lightgray color with 0.3 opacity

That’s it! With just a few quick additions, the visualization now looks like this:

For even more information about what you can do upon selection, check out Selected and Unselected Glyphs.

Adding Hover Actions

So the ability to select specific player data points that seem of interest in my scatter plot is implemented, but what if you want to quickly see what individual players a glyph represents? One option is to use Bokeh’s HoverTool() to show a tooltip when the cursor crosses paths with a glyph. All you need to do is append the following to the code snippet above:

# Bokeh Library
from bokeh.models import HoverTool

# Format the tooltip
tooltips = [
            ('Player','@name'),
            ('Three-Pointers Made', '@play3PM'),
            ('Three-Pointers Attempted', '@play3PA'),
            ('Three-Point Percentage','@pct3PM{00.0%}'),
           ]

# Add the HoverTool to the figure
fig.add_tools(HoverTool(tooltips=tooltips))

# Visualize
show(fig)

The HoverTool() is slightly different than the selection tools you saw above in that it has properties, specifically tooltips.

First, you can configure a formatted tooltip by creating a list of tuples containing a description and reference to the ColumnDataSource. This list was passed as input to the HoverTool() and then simply added to the figure using add_tools(). Here’s what happened:

Notice the addition of the Hover button to the toolbar, which can be toggled on and off.

If you want to even further emphasize the players on hover, Bokeh makes that possible with hover inspections. Here is a slightly modified version of the code snippet that added the tooltip:

# Format the tooltip
tooltips = [
            ('Player','@name'),
            ('Three-Pointers Made', '@play3PM'),
            ('Three-Pointers Attempted', '@play3PA'),
            ('Three-Point Percentage','@pct3PM{00.0%}'),
           ]

# Configure a renderer to be used upon hover
hover_glyph = fig.circle(x='play3PA', y='pct3PM', source=three_takers_cds,
                         size=15, alpha=0,
                         hover_fill_color='black', hover_alpha=0.5)

# Add the HoverTool to the figure
fig.add_tools(HoverTool(tooltips=tooltips, renderers=[hover_glyph]))

# Visualize
show(fig)

This is done by creating a completely new glyph, in this case circles instead of squares, and assigning it to hover_glyph. Note that the initial opacity is set to zero so that it is invisible until the cursor is touching it. The properties that appear upon hover are captured by setting hover_alpha to 0.5 along with the hover_fill_color.

Now you will see a small black circle appear over the original square when hovering over the various markers:

To further explore the capabilities of the HoverTool(), see the HoverTool and Hover Inspections guides.

Linking Axes and Selections

Linking is the process of syncing elements of different visualizations within a layout. For instance, maybe you want to link the axes of multiple plots to ensure that if you zoom in on one it is reflected on another. Let’s see how it is done.

For this example, the visualization will be able to pan to different segments of a team’s schedule and examine various game stats. Each stat will be represented by its own plot in a two-by-two gridplot() .

The data can be collected from the team_stats DataFrame, selecting the Philadelphia 76ers as the team of interest:

# Isolate relevant data
phi_gm_stats = (team_stats[(team_stats['teamAbbr'] == 'PHI') & 
                           (team_stats['seasTyp'] == 'Regular')]
                .loc[:, ['gmDate', 
                         'teamPTS', 
                         'teamTRB', 
                         'teamAST', 
                         'teamTO', 
                         'opptPTS',]]
                .sort_values('gmDate'))

# Add game number
phi_gm_stats['game_num'] = range(1, len(phi_gm_stats)+1)

# Derive a win_loss column
win_loss = []
for _, row in phi_gm_stats.iterrows():

    # If the 76ers score more points, it's a win
    if row['teamPTS'] > row['opptPTS']:
        win_loss.append('W')
    else:
        win_loss.append('L')

# Add the win_loss data to the DataFrame
phi_gm_stats['winLoss'] = win_loss

Here are the results of the 76ers’ first 5 games:

>>>

>>> phi_gm_stats.head()
        gmDate  teamPTS  teamTRB  teamAST  teamTO  opptPTS  game_num winLoss
10  2017-10-18      115       48       25      17      120         1       L
39  2017-10-20       92       47       20      17      102         2       L
52  2017-10-21       94       41       18      20      128         3       L
80  2017-10-23       97       49       25      21       86         4       W
113 2017-10-25      104       43       29      16      105         5       L

Start by importing the necessary Bokeh libraries, specifying the output parameters, and reading the data into a ColumnDataSource:

# Bokeh Libraries
from bokeh.plotting import figure, show
from bokeh.io import output_file
from bokeh.models import ColumnDataSource, CategoricalColorMapper, Div
from bokeh.layouts import gridplot, column

# Output to file
output_file('phi-gm-linked-stats.html',
                title='76ers Game Log')

# Store the data in a ColumnDataSource
gm_stats_cds = ColumnDataSource(phi_gm_stats)

Each game is represented by a column, and will be colored green if the result was a win and red for a loss. To accomplish this, Bokeh’s CategoricalColorMapper can be used to map the data values to specified colors:

# Create a CategoricalColorMapper that assigns a color to wins and losses
win_loss_mapper = CategoricalColorMapper(factors = ['W', 'L'], 
                                         palette=['green', 'red'])

For this use case, a list specifying the categorical data values to be mapped is passed to factors and a list with the intended colors to palette. For more on the CategoricalColorMapper, see the Colors section of Handling Categorical Data on Bokeh’s User Guide.

There are four stats to visualize in the two-by-two gridplot: points, assists, rebounds, and turnovers. In creating the four figures and configuring their respective charts, there is a lot of redundancy in the properties. So to streamline the code a for loop can be used:

# Create a dict with the stat name and its corresponding column in the data
stat_names = {'Points': 'teamPTS',
              'Assists': 'teamAST',
              'Rebounds': 'teamTRB',
              'Turnovers': 'teamTO',}

# The figure for each stat will be held in this dict
stat_figs = {}

# For each stat in the dict
for stat_label, stat_col in stat_names.items():

    # Create a figure
    fig = figure(y_axis_label=stat_label, 
                 plot_height=200, plot_width=400,
                 x_range=(1, 10), tools=['xpan', 'reset', 'save'])

    # Configure vbar
    fig.vbar(x='game_num', top=stat_col, source=gm_stats_cds, width=0.9, 
             color=dict(field='winLoss', transform=win_loss_mapper))

    # Add the figure to stat_figs dict
    stat_figs[stat_label] = fig

As you can see, the only parameters that needed to be adjusted were the y-axis-label of the figure and the data that will dictate top in the vbar. These values were easily stored in a dict that was iterated through to create the figures for each stat.

You can also see the implementation of the CategoricalColorMapper in the configuration of the vbar glyph. The color property is passed a dict with the field in the ColumnDataSource to be mapped and the name of the CategoricalColorMapper created above.

The initial view will only show the first 10 games of the 76ers’ season, so there needs to be a way to pan horizontally to navigate through the rest of the games in the season. Thus configuring the toolbar to have an xpan tool allows panning throughout the plot without having to worry about accidentally skewing the view along the vertical axis.

Now that the figures are created, gridplot can be setup by referencing the figures from the dict created above:

# Create layout
grid = gridplot([[stat_figs['Points'], stat_figs['Assists']], 
                [stat_figs['Rebounds'], stat_figs['Turnovers']]])

Linking the axes of the four plots is as simple as setting the x_range of each figure equal to one another:

# Link together the x-axes
stat_figs['Points'].x_range = \
    stat_figs['Assists'].x_range = \
    stat_figs['Rebounds'].x_range = \
    stat_figs['Turnovers'].x_range

To add a title bar to the visualization, you could have tried to do this on the points figure, but it would have been limited to the space of that figure. Therefore, a nice trick is to use Bokeh’s ability to interpret HTML to insert a Div element that contains the title information. Once that is created, simply combine that with the gridplot() in a column layout:

# Add a title for the entire visualization using Div
html = """<h3>Philadelphia 76ers Game Log</h3>
<b><i>2017-18 Regular Season</i>
<br>
</b><i>Wins in green, losses in red</i>
"""
sup_title = Div(text=html)

# Visualize
show(column(sup_title, grid))

Putting all the pieces together results in the following:

Similarly you can easily implement linked selections, where a selection on one plot will be reflected on others.

To see how this works, the next visualization will contain two scatter plots: one that shows the 76ers’ two-point versus three-point field goal percentage and the other showing the 76ers’ team points versus opponent points on a game-by-game basis.

The goal is to be able to select data points on the left-side scatter plot and quickly be able to recognize if the corresponding datapoint on the right scatter plot is a win or loss.

The DataFrame for this visualization is very similar to that from the first example:

# Isolate relevant data
phi_gm_stats_2 = (team_stats[(team_stats['teamAbbr'] == 'PHI') & 
                             (team_stats['seasTyp'] == 'Regular')]
                  .loc[:, ['gmDate', 
                           'team2P%', 
                           'team3P%', 
                           'teamPTS', 
                           'opptPTS']]
                  .sort_values('gmDate'))

# Add game number
phi_gm_stats_2['game_num'] = range(1, len(phi_gm_stats_2) + 1)

# Derive a win_loss column
win_loss = []
for _, row in phi_gm_stats_2.iterrows():

    # If the 76ers score more points, it's a win
    if row['teamPTS'] > row['opptPTS']:
        win_loss.append('W')
    else:
        win_loss.append('L')

# Add the win_loss data to the DataFrame
phi_gm_stats_2['winLoss'] = win_loss

Here’s what the data looks like:

>>>

>>> phi_gm_stats_2.head()
        gmDate  team2P%  team3P%  teamPTS  opptPTS  game_num winLoss
10  2017-10-18   0.4746   0.4286      115      120         1       L
39  2017-10-20   0.4167   0.3125       92      102         2       L
52  2017-10-21   0.4138   0.3333       94      128         3       L
80  2017-10-23   0.5098   0.3750       97       86         4       W
113 2017-10-25   0.5082   0.3333      104      105         5       L

The code to create the visualization is as follows:

# Bokeh Libraries
from bokeh.plotting import figure, show
from bokeh.io import output_file
from bokeh.models import ColumnDataSource, CategoricalColorMapper, NumeralTickFormatter
from bokeh.layouts import gridplot

# Output inline in the notebook
output_file('phi-gm-linked-selections.html',
            title='76ers Percentages vs. Win-Loss')

# Store the data in a ColumnDataSource
gm_stats_cds = ColumnDataSource(phi_gm_stats_2)

# Create a CategoricalColorMapper that assigns specific colors to wins and losses
win_loss_mapper = CategoricalColorMapper(factors = ['W', 'L'], palette=['Green', 'Red'])

# Specify the tools
toolList = ['lasso_select', 'tap', 'reset', 'save']

# Create a figure relating the percentages
pctFig = figure(title='2PT FG % vs 3PT FG %, 2017-18 Regular Season',
                plot_height=400, plot_width=400, tools=toolList,
                x_axis_label='2PT FG%', y_axis_label='3PT FG%')

# Draw with circle markers
pctFig.circle(x='team2P%', y='team3P%', source=gm_stats_cds, 
              size=12, color='black')

# Format the y-axis tick labels as percenages
pctFig.xaxis[0].formatter = NumeralTickFormatter(format='00.0%')
pctFig.yaxis[0].formatter = NumeralTickFormatter(format='00.0%')

# Create a figure relating the totals
totFig = figure(title='Team Points vs Opponent Points, 2017-18 Regular Season',
                plot_height=400, plot_width=400, tools=toolList,
                x_axis_label='Team Points', y_axis_label='Opponent Points')

# Draw with square markers
totFig.square(x='teamPTS', y='opptPTS', source=gm_stats_cds, size=10,
              color=dict(field='winLoss', transform=win_loss_mapper))

# Create layout
grid = gridplot([[pctFig, totFig]])

# Visualize
show(grid)

This is a great illustration of the power in using a ColumnDataSource. As long as the glyph renderers (in this case, the circle glyphs for the percentages, and square glyphs for the wins and losses) share the same ColumnDataSource, then the selections will be linked by default.

Here’s how it looks in action, where you can see selections made on either figure will be reflected on the other:

By selecting a random sample of data points in the upper right quadrant of the left scatter plot, those corresponding to both high two-point and three-point field goal percentage, the data points on the right scatter plot are highlighted.

Similarly, selecting data points on the right scatter plot that correspond to losses tend to be further to the lower left, lower shooting percentages, on the left scatter plot.

All the details on linking plots can be found at Linking Plots in the Bokeh User Guide.

Highlighting Data Using the Legend

That brings us to the final interactivity example in this tutorial: interactive legends.

In the Drawing Data With Glyphs section, you saw how easy it is to implement a legend when creating your plot. With the legend in place, adding interactivity is merely a matter of assigning a click_policy. Using a single line of code, you can quickly add the ability to either hide or mute data using the legend.

In this example, you’ll see two identical scatter plots comparing the game-by-game points and rebounds of LeBron James and Kevin Durant. The only difference will be that one will use a hide as its click_policy, while the other uses mute.

The first step is to configure the output and set up the data, creating a view for each player from the player_stats DataFrame:

# Bokeh Libraries
from bokeh.plotting import figure, show
from bokeh.io import output_file
from bokeh.models import ColumnDataSource, CDSView, GroupFilter
from bokeh.layouts import row

# Output inline in the notebook
output_file('lebron-vs-durant.html',
            title='LeBron James vs. Kevin Durant')

# Store the data in a ColumnDataSource
player_gm_stats = ColumnDataSource(player_stats)

# Create a view for each player
lebron_filters = [GroupFilter(column_name='playFNm', group='LeBron'),
                  GroupFilter(column_name='playLNm', group='James')]
lebron_view = CDSView(source=player_gm_stats,
                      filters=lebron_filters)

durant_filters = [GroupFilter(column_name='playFNm', group='Kevin'),
                  GroupFilter(column_name='playLNm', group='Durant')]
durant_view = CDSView(source=player_gm_stats,
                      filters=durant_filters)

Before creating the figures, the common parameters across the figure, markers, and data can be consolidated into dictionaries and reused. Not only does this save redundancy in the next step, but it provides an easy way to tweak these parameters later if need be:

# Consolidate the common keyword arguments in dicts
common_figure_kwargs = {
    'plot_width': 400,
    'x_axis_label': 'Points',
    'toolbar_location': None,
}
common_circle_kwargs = {
    'x': 'playPTS',
    'y': 'playTRB',
    'source': player_gm_stats,
    'size': 12,
    'alpha': 0.7,
}
common_lebron_kwargs = {
    'view': lebron_view,
    'color': '#002859',
    'legend': 'LeBron James'
}
common_durant_kwargs = {
    'view': durant_view,
    'color': '#FFC324',
    'legend': 'Kevin Durant'
}

Now that the various properties are set, the two scatter plots can be built in a much more concise fashion:

# Create the two figures and draw the data
hide_fig = figure(**common_figure_kwargs,
                  title='Click Legend to HIDE Data', 
                  y_axis_label='Rebounds')
hide_fig.circle(**common_circle_kwargs, **common_lebron_kwargs)
hide_fig.circle(**common_circle_kwargs, **common_durant_kwargs)

mute_fig = figure(**common_figure_kwargs, title='Click Legend to MUTE Data')
mute_fig.circle(**common_circle_kwargs, **common_lebron_kwargs,
                muted_alpha=0.1)
mute_fig.circle(**common_circle_kwargs, **common_durant_kwargs,
                muted_alpha=0.1)

Note that mute_fig has an extra parameter called muted_alpha. This parameter controls the opacity of the markers when mute is used as the click_policy.

Finally, the click_policy for each figure is set, and they are shown in a horizontal configuration:

# Add interactivity to the legend
hide_fig.legend.click_policy = 'hide'
mute_fig.legend.click_policy = 'mute'

# Visualize
show(row(hide_fig, mute_fig))

Once the legend is in place, all you have to do is assign either hide or mute to the figure’s click_policy property. This will automatically turn your basic legend into an interactive legend.

Also note that, specifically for mute, the additional property of muted_alpha was set in the respective circle glyphs for LeBron James and Kevin Durant. This dictates the visual effect driven by the legend interaction.

For more on all things interaction in Bokeh, Adding Interactions in the Bokeh User Guide is a great place to start.

Summary and Next Steps

Congratulations! You’ve made it to the end of this tutorial.

You should now have a great set of tools to start turning your data into beautiful interactive visualizations using Bokeh.

You learned how to:

Configure your script to render to either a static HTML file or Jupyter Notebook
Instantiate and customize the figure() object
Build your visualization using glyphs
Access and filter your data with the ColumnDataSource
Organize multiple plots in grid and tabbed layouts
Add different forms of interaction, including selections, hover actions, linking, and interactive legends

To explore even more of what Bokeh is capable of, the official Bokeh User Guide is an excellent place to dig into some more advanced topics. I’d also recommend checking out Bokeh’s Gallery for tons of examples and inspiration.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python Community Interview With Kenneth Reitz

2018-11-14T14:00:00+00:00

This week, I’m excited to be interviewing the prolific Kenneth Reitz!

Kenneth is the author of the extremely popular requests and pipenv libraries. Join us as we talk about his latest projects and the most challenging code he’s written to date.

Ricky: Let’s start at the beginning… How did you get into programming, and when did you start using Python?

Kenneth: I started programming at a young age. My dad was a programmer, and I taught myself BASIC and C (with this help) at the age of 9. I started using Python in college, when I took my first CS course. Shortly after, I dropped out and learned many other programming languages, but I always kept coming back to Python.

Ricky: Congratulations on your new job with Digital Ocean. You’re the senior member of the Developer Relations team. How are you finding the change in role from your previous job at Heroku, and what can we expect from Digital Ocean moving forward in the Python space?

Kenneth: Thanks! I’m really enjoying the new role and the opportunity to serve the entire development community, not just the Python community. However, my latest work, Responder, has been a Digital Ocean project, so there’s room to expect more from us in the Python space 😊

Ricky: You are, of course, most famous for writing the extremely popular requests library and the new pipenv library. Python.org now recommends the use of pipenv for dependency management. How has the community received pipenv? Have you seen much resistance from the community with developers preferring to stick to venv or older methods of dependency management?

Kenneth: The community has received pipenv very well, and even companies like GitHub are using its standards for security vulnerability scanning. I haven’t seen much resistance from the community at all, aside from some hatred on reddit. It took me a while to realize that /r/python doesn’t represent the Python community as much as it represents redditors who use Python.

Ricky: Now hitting 300 million downloads on your requests library is cool and all, but as a guitarist, what I’m more excited about is your latest project PyTheory. Can you tell us a little about it and your plans for the project going forward?

Kenneth: PyTheory is a very interesting library that attempts to encapsulate all known musical systems into a library. Currently, there is one system: Western. It can render all the different scales for the Western system, programmatically, and tell you the pitches of the notes (either in decimal or symbolic notation). In addition, there are fretboards and chord charts, so you can specify a custom tuning for your guitar, and generate chord charts with it. It’s very abstract.

Definitely the most challenging thing I’ve ever written.

Ricky: So after recently ditching your Mac for a PC, and turning to Microsoft’s (amazing) VS Code for your Python development, are you happy and proud to be a Windows user? For those who may be reading who haven’t used Windows since Windows ‘95, what are they missing?

Kenneth: I love the Mac and prefer it to Windows. I’m just currently bored with my setup and decided to challenge myself by running Windows. I’m very happy and productive, though. Feels right at home.

Windows isn’t what it used to be. It’s a real solid piece of operating system now. I’m running it on my iMac Pro, and it hums like a dream.

Ricky: I know you’re a keen photographer. How long have you been into it, and what’s your favorite photo you’ve taken? Do you have any other hobbies and interests, aside from Python?

Kenneth: I’ve been into photography seriously for about 10 years. My favorite photo I’ve ever taken is probably this one. It was taken on a film camera, after I had a migraine for several weeks, and it was my first time being able to walk outside.

Ricky: Finally, any parting words of wisdom? Anything you’d like to share and/or plug?

Kenneth: Responder! My new web service framework for Python. It’s ASGI, it’s familiar looking, it’s fast, and it’s easier to use than Flask! Great for building APIs. Check it out!

Thank you, Kenneth, for joining me this week. It’s been great to watch the development of Responder happen in realtime on Twitter. You can follow its development or raise an issue here.

As always, if there is someone you would like me to interview in the future, reach out to me in the comments below, or send me a message on Twitter.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

How to Publish an Open-Source Python Package to PyPI

2018-11-12T14:00:00+00:00

Python is famous for coming with batteries included. Sophisticated capabilities are available in the standard library. You can find modules for working with sockets, parsing CSV, JSON, and XML files, and working with files and file paths.

However great the packages included with Python are, there are many fantastic projects available outside the standard library. These are most often hosted at the Python Packaging Index (PyPI), historically known as the Cheese Shop. At PyPI, you can find everything from Hello World to advanced deep learning libraries.

In this tutorial, you’ll cover how to upload your own package to PyPI. While getting your project published is easier than it used to be, there are still a few steps involved.

You’ll learn how to:

Prepare your Python package for publication
Think about versioning
Upload your package to PyPI

Throughout this tutorial, we’ll use a simple example project: a reader package that can be used to read Real Python tutorials. The first section introduces this project.

A Small Python Package

This section will describe a small Python package that we’ll use as an example that can be published to PyPI. If you already have a package that you want to publish, feel free to skim this section and join up again at the beginning of the next section.

The package that we’ll use is called reader and is an application that can be used to download and read Real Python articles. If you want to follow along, you can get the full source code from our GitHub repository.

Note: The source code as shown and explained below is a simplified, but fully functional, version of the Real Python feed reader. Compared to the package published on PyPI and GitHub, this version lacks some error handling and extra options.

First, have a look at the directory structure of reader. The package lives completely inside a directory that is also named reader:

reader/
│
├── reader/
│   ├── config.txt
│   ├── feed.py
│   ├── __init__.py
│   ├── __main__.py
│   └── viewer.py
│
├── tests/
│   ├── test_feed.py
│   └── test_viewer.py
│
├── MANIFEST.in
├── README.md
└── setup.py

The source code of the package is in a reader subdirectory together with a configuration file. There are a few tests in a separate subdirectory. The tests will not be covered here, but you can find them in the GitHub repository. To learn more about testing, see Anthony Shaw’s great tutorial on Getting Started With Testing in Python.

If you’re working with your own package, you may use a different structure or have other files in your package directory. Our Python Application Layouts reference discusses several different options. The instructions in this guide will work independently of the layout you use.

In the rest of this section, you’ll see how the reader package works. In the next section, you’ll get a closer look at the special files, including setup.py, README.md, and MANIFEST.in, that are needed to publish your package.

Using the Real Python Reader

reader is a very basic web feed reader that can download the latest Real Python articles from the Real Python feed.

Here is an example of using the reader to get the list of the latest articles:

$ python -m reader
The latest tutorials from Real Python (https://realpython.com/)
  0 How to Publish an Open-Source Python Package to PyPI
  1 Python "while" Loops (Indefinite Iteration)
  2 Writing Comments in Python (Guide)
  3 Setting Up Python for Machine Learning on Windows
  4 Python Community Interview With Michael Kennedy
  5 Practical Text Classification With Python and Keras
  6 Getting Started With Testing in Python
  7 Python, Boto3, and AWS S3: Demystified
  8 Python's range() Function (Guide)
  9 Python Community Interview With Mike Grouchy
 10 How to Round Numbers in Python
 11 Building and Documenting Python REST APIs With Flask and Connexion – Part 2
 12 Splitting, Concatenating, and Joining Strings in Python
 13 Image Segmentation Using Color Spaces in OpenCV + Python
 14 Python Community Interview With Mahdi Yusuf
 15 Absolute vs Relative Imports in Python
 16 Top 10 Must-Watch PyCon Talks
 17 Logging in Python
 18 The Best Python Books
 19 Conditional Statements in Python

Notice that each article is numbered. To read one particular article, you use the same command but include the number of the article as well. For instance, to read How to Publish an Open-Source Python Package to PyPI, you add 0 to the command:

$ python -m reader 0
# How to Publish an Open-Source Python Package to PyPI

Python is famous for coming with batteries included. Sophisticated
capabilities are available in the standard library. You can find modules
for working with sockets, parsing CSV, JSON, and XML files, and
working with files and file paths.

However great the packages included with Python are, there are many
fantastic projects available outside the standard library. These are
most often hosted at the Python Packaging Index (PyPI), historically
known as the Cheese Shop. At PyPI, you can find everything from Hello
World to advanced deep learning libraries.

[... The full text of the article ...]

This prints the full article to the console using the Markdown text format.

Note: python -m is used to run a library module or package instead of a script. If you run a package, the contents of the file __main__.py will be executed. See Different Ways of Calling a Package for more info.

By changing the article number, you can read any of the available articles.

A Quick Look at the Code

The details of how reader works are not important for the purpose of this tutorial. However, if you are interested in seeing the implementation, you can expand the sections below. The package consists of five files:

config.txt is a configuration file used to specify the URL of the feed of Real Python tutorials. It’s a text file that can be read by the configparser standard library:

# config.txt

[feed]
url = https://realpython.com/atom.xml

In general, such a config file contains key-value pairs separated into sections. This particular file contains only one section (feed) and one key (url).

Note: A configuration file is probably overkill for this simple package. We include it here for demonstration purposes.

The first source code file we’ll look at is __main__.py. The double underscores indicate that this file has a special meaning in Python. Indeed, when running a package as a script with -m as above, Python executes the contents of the __main__.py file.

In other words, __main__.py acts as the entry point of our program and takes care of the main flow, calling other parts as needed:

# __main__.py

from configparser import ConfigParser
from importlib import resources  # Python 3.7+
import sys

from reader import feed
from reader import viewer

def main():
    """Read the Real Python article feed"""
    # Read URL of the Real Python feed from config file
    cfg = ConfigParser()
    cfg.read_string(resources.read_text("reader", "config.txt"))
    url = cfg.get("feed", "url")

    # If an article ID is given, show the article
    if len(sys.argv) > 1:
        article = feed.get_article(url, sys.argv[1])
        viewer.show(article)

    # If no ID is given, show a list of all articles
    else:
        site = feed.get_site(url)
        titles = feed.get_titles(url)
        viewer.show_list(site, titles)

if __name__ == "__main__":
    main()

Notice that main() is called on the last line. If we do not call main(), then our program would not do anything. As you saw earlier, the program can either list all articles or print one specific article. This is handled by the if-else inside main().

To read the URL to the feed from the configuration file, we use configparser and importlib.resources. The latter is used to import non-code (or resource) files from a package without having to worry about the full file path. It is especially helpful when publishing packages to PyPI where resource files might end up inside binary archives.

importlib.resources became a part of the standard library in Python 3.7. If you are using an older version of Python, you can use importlib_resources instead. This is a backport compatible with Python 2.7, and 3.4 and above. importlib_resources can be installed from PyPI:

$ pip install importlib_resources

See Barry Warzaw’s presentation at PyCon 2018 for more information.

The next file is __init__.py. Again, the double underscores in the filename tell us that this is a special file. __init__.py represents the root of your package. It should usually be kept quite simple, but it’s a good place to put package constants, documentation, and so on:

# __init__.py

# Version of the realpython-reader package
__version__ = "1.0.0"

The special variable __version__ is a convention in Python for adding version numbers to your package. It was introduced in PEP 396. We’ll talk more about versioning later.

Variables defined in __init__.py become available as variables in the package namespace:

>>>

>>> import reader
>>> reader.__version__
'1.0.0'

You should define the __version__ variable in your own packages as well.

Looking at __main__.py, you’ll see that two modules, feed and viewer, are imported and used to read from the feed and show the results. These modules do most of the actual work.

First consider feed.py. This file contains functions for reading from a web feed and parsing the result. Luckily there are already great libraries available to do this. feed.py depends on two modules that are already available on PyPI: feedparser and html2text.

feed.py contains several functions. We’ll discuss them one at a time.

To avoid reading from the web feed more than necessary, we first create a function that remembers the feed the first time it’s read:

 1 # feed.py
 2 
 3 import feedparser
 4 import html2text
 5 
 6 _CACHED_FEEDS = dict()
 7 
 8 def _feed(url):
 9     """Only read a feed once, by caching its contents"""
10     if url not in _CACHED_FEEDS:
11         _CACHED_FEEDS[url] = feedparser.parse(url)
12     return _CACHED_FEEDS[url]

feedparser.parse() reads a feed from the web and returns it in a structure that looks like a dictionary. To avoid downloading the feed more than once, it’s stored in _CACHED_FEEDS and reused for later calls to _feed(). Both _CACHED_FEEDS and _feed() are prefixed by an underscore to indicate that they are support objects not meant to be used directly.

We can get some basic information about the feed by looking in the .feed metadata. The following function picks out the title and link to the web site containing the feed:

14 def get_site(url):
15     """Get name and link to web site of the feed"""
16     info = _feed(url).feed
17     return f"{info.title} ({info.link})"

In addition to .title and .link, attributes like .subtitle, .updated, and .id are also available.

The articles available in the feed can be found inside the .entries list. Article titles can be found with a list comprehension:

19 def get_titles(url):
20     """List titles in feed"""
21     articles = _feed(url).entries
22     return [a.title for a in articles]

.entries lists the articles in the feed sorted chronologically, so that the newest article is .entries[0].

In order to get the contents of one article, we use its index in the .entries list as an article ID:

24 def get_article(url, article_id):
25     """Get article from feed with the given ID"""
26     articles = _feed(url).entries
27     article = articles[int(article_id)]
28     html = article.content[0].value
29     text = html2text.html2text(html)
30     return f"# {article.title}\n\n{text}"

After picking the correct article out of the .entries list, we find the text of the article as HTML on line 28. Next, html2text does a decent job of translating the HTML into much more readable text. As the HTML doesn’t contain the title of the article, the title is added before returning.

The final module is viewer.py. At the moment, it consists of two very simple functions. In practice, we could have used print() directly in __main__.py instead of calling viewer functions. However, having the functionality split off makes it easier to replace it later with something more advanced. Maybe we could add a GUI interface in a later version?

viewer.py contains two functions:

# viewer.py

def show(article):
    """Show one article"""
    print(article)

def show_list(site, titles):
    """Show list of articles"""
    print(f"The latest tutorials from {site}")
    for article_id, title in enumerate(titles):
        print(f"{article_id:>3} {title}")

show() simply prints one article to the console, while show_list() prints a list of titles. The latter also creates article IDs that are used when choosing to read one particular article.

Different Ways of Calling a Package

One challenge when your projects grow in complexity is communicating to the user how to use your project. Since the package consists of four different source code files, how does the user know which file to call to run reader?

The python interpreter program has an -m option that allows you to specify a module name instead of a file name. For instance, if you have a script called hello.py, the following two commands are equivalent:

$ python hello.py
Hi there!

$ python -m hello
Hi there!

One advantage of the latter is that it allows you to call modules that are built into Python as well. One example is calling antigravity:

$ python -m antigravity
Created new window in existing browser session.

Another advantage of using -m is that it works for packages as well as modules. As you saw earlier, you can call the reader package with -m:

$ python -m reader
[...]

Since reader is a package, the name only refers to a directory. How does Python decide which code inside that directory to run? It looks for a file named __main__.py. If such a file exists, it is executed. If __main__.py does not exist, then an error message is printed:

$ python -m math
python: No code object available for math

In this example, you see that the math standard library has not defined a __main__.py file.

If you are creating a package that is supposed to be executed, you should include a __main__.py file. Later, you’ll see how you can also create entry points to your package that will behave like regular programs.

Preparing Your Package for Publication

Now you’ve got a package you want to publish, or maybe you copied our package. Which steps are necessary before you can upload the package to PyPI?

Naming Your Package

The first—and possibly the hardest—step is to come up with a good name for your package. All packages on PyPI need to have unique names. With more than 150,000 packages already on PyPI, chances are that your favorite name is already taken.

You might need to brainstorm and do some research to find the perfect name. Use the PyPI search to check if a name is already taken. The name that you come up with will be visible on PyPI.

To make the reader package easier to find on PyPI, we give it a more descriptive name and call it realpython-reader. The same name will be used to install the package using pip:

$ pip install realpython-reader

Even though we use realpython-reader as the PyPI name, the package is still called reader when it’s imported:

>>>

>>> import reader
>>> help(reader)

>>> from reader import feed
>>> feed.get_titles()
['How to Publish an Open-Source Python Package to PyPI', ...]

As you see, you can use different names for your package on PyPI and when importing. However, if you use the same name or very similar names, then it will be easier for your users.

Configuring Your Package

In order for your package to be uploaded to PyPI, you need to provide some basic information about it. This information is typically provided in the form of a setup.py file. There are initiatives that try to simplify this collection of information. At the moment though, setup.py is the only fully supported way of providing information about your package.

The setup.py file should be placed in the top folder of your package. A fairly minimal setup.py for reader looks like this:

import pathlib
from setuptools import setup

# The directory containing this file
HERE = pathlib.Path(__file__).parent

# The text of the README file
README = (HERE / "README.md").read_text()

# This call to setup() does all the work
setup(
    name="realpython-reader",
    version="1.0.0",
    description="Read the latest Real Python tutorials",
    long_description=README,
    long_description_content_type="text/markdown",
    url="https://github.com/realpython/reader",
    author="Real Python",
    author_email="office@realpython.com",
    license="MIT",
    classifiers=[
        "License :: OSI Approved :: MIT License",
        "Programming Language :: Python :: 3",
        "Programming Language :: Python :: 3.7",
    ],
    packages=["reader"],
    include_package_data=True,
    install_requires=["feedparser", "html2text"],
    entry_points={
        "console_scripts": [
            "realpython=reader.__main__:main",
        ]
    },
)

We will only cover some of the options available in setuptools here. The documentation does a good job of going into all the detail.

The parameters that are 100% necessary in the call to setup() are the following:

name: the name of your package as it will appear on PyPI
version: the current version of your package
packages: the packages and subpackages containing your source code

We will talk more about versions later. The packages parameter takes a list of packages. In our example, there is only one package: reader.

You also need to specify any subpackages. In more complicated projects, there might be many packages to list. To simplify this job, setuptools includes find_packages(), which does a good job of discovering all your subpackages. You could have used find_packages() in the reader project as follows:

from setuptools import find_packages, setup

setup(
    ...
    packages=find_packages(exclude=("tests",)),
    ...
)

While only name, version, and packages are required, your package becomes much easier to find on PyPI if you add some more information. Have a look at the realpython-reader page on PyPI and compare the information with setup.py above. All the information comes from setup.py and README.md.

realpython-reader package at PyPI"/>

The last two parameters to setup() deserve special mention:

install_requires is used to list any dependencies your package has to third party libraries. The reader depends on feedparser and html2text, so they should be listed here.
entry_points is used to create scripts that call a function within your package. In our example, we create a new script realpython that calls main() within the reader/__main__.py file.

For another example of a typical setup file, see Kenneth Reitz’s setup.py repository on GitHub.

Documenting Your Package

Before releasing your package to the world, you should add some documentation. Depending on your package, the documentation can be as small as a simple README file, or as big as a full web page with tutorials, example galleries, and an API reference.

At a minimum, you should include a README file with your project. A good README should quickly describe your project, as well as tell your users how to install and use your package. Typically, you want to include your README as the long_description argument to setup(). This will display your README on PyPI.

Traditionally, PyPI has used reStructuredText for package documentation. However, since March 2018 Markdown has also been supported.

Outside of PyPI, Markdown is more widely supported than reStructuredText. If you don’t need any of the special features of reStructuredText, you’ll be better off keeping your README in Markdown. Note that you should use the setup() parameter long_description_content_type to tell PyPI which format you are using. Valid values are text/markdown, text/x-rst, and text/plain.

For bigger projects, you might want to offer more documentation than can reasonably fit in a single file. In that case, you can use sites like GitHub or Read the Docs, and link to the documentation using the url parameter. In the setup.py example above, url is used to link to the reader GitHub repository.

Versioning Your Package

Your package needs to have a version, and PyPI will only let you do one upload of a particular version for a package. In other words, if you want to update your package on PyPI, you need to increase the version number first. This is a good thing, as it guarantees reproducibility: two systems with the same version of a package should behave the same.

There are many different schemes that can be used for your version number. For Python projects, PEP 440 gives some recommendations. However, in order to be flexible, that PEP is complicated. For a simple project, stick with a simple versioning scheme.

Semantic versioning is a good default scheme to use. The version number is given as three numerical components, for instance 0.1.2. The components are called MAJOR, MINOR, and PATCH, and there are simple rules about when to increment each component:

Increment the MAJOR version when you make incompatible API changes.

Increment the MINOR version when you add functionality in a backwards-compatible manner.

Increment the PATCH version when you make backwards-compatible bug fixes. (Source)

You may need to specify the version in different files inside your project. In the reader project, we specified the version both in setup.py and in reader/__init__.py. To make sure the version numbers are kept consistent, you can use a tool called Bumpversion.

You can install Bumpversion from PyPI:

$ pip install bumpversion

To increment the MINOR version of reader, you would do something like this:

$ bumpversion --current-version 1.0.0 minor setup.py reader/__init__.py

This would change the version number from 1.0.0 to 1.1.0 in both setup.py and reader/__init__.py. To simplify the command, you can also give most of the information in a configuration file. See the Bumpversion documentation for details.

Adding Files to Your Package

Sometimes, you’ll have files inside your package that are not source code files. Examples include data files, binaries, documentation, and—as we have in this project—configuration files.

To tell setup() to include such files, you use a manifest file. For many projects, you don’t need to worry about the manifest, as setup() creates one that includes all code files as well as README files.

If you need to change the manifest, you create a manifest template which must be named MANIFEST.in. This file specifies rules for what to include and exclude:

include reader/*.txt

This example will include all .txt files in the reader directory, which in effect is the configuration file. See the documentation for a list of available rules.

In addition to creating MANIFEST.in, you also need to tell setup() to copy these non-code files. This is done by setting the include_package_data argument to True:

setup(
    ...
    include_package_data=True,
    ...
)

The include_package_data argument controls whether non-code files are copied when your package is installed.

Publishing to PyPI

Your package is finally ready to meet the world outside your computer! In this section, you’ll see how to actually upload your package to PyPI.

If you don’t already have an account on PyPI, now is the time to create one: register your account on PyPI. While you’re at it, you should also register an account on TestPyPI. TestPyPI is very useful, as you can try all the steps of publishing a package without any consequences if you mess up.

To upload your package to PyPI, you’ll use a tool called Twine. You can install Twine using Pip as usual:

$ pip install twine

Using Twine is quite simple, and you will soon see how to use it to check and publish your package.

Building Your Package

Packages on PyPI are not distributed as plain source code. Instead, they are wrapped into distribution packages. The most common formats for distribution packages are source archives and Python wheels.

A source archive consists of your source code and any supporting files wrapped into one tar file. Similarly, a wheel is essentially a zip archive containing your code. In contrast to the source archive, the wheel includes any extensions ready to use.

To create a source archive and a wheel for your package, you can run the following command:

$ python setup.py sdist bdist_wheel

This will create two files in a newly created dist directory, a source archive and a wheel:

reader/
│
└── dist/
    ├── realpython_reader-1.0.0-py3-none-any.whl
    └── realpython-reader-1.0.0.tar.gz

Note: On Windows, the source archive will be a .zip file by default. You can choose the format of the source archive with the --format command line option.

You might wonder how setup.py knows what to do with the sdist and bdist_wheel arguments. If you look back to how setup.py was implemented, there is no mention of sdist, bdist_wheel, or any other command line arguments.

All the command line arguments are instead implemented in the upstream distutils standard library. You can list all available arguments by adding the --help-commands option:

$ python setup.py --help-commands
Standard commands:
  build             build everything needed to install
  build_py          "build" pure Python modules (copy to build directory)
  build_ext         build C/C++ and Cython extensions (compile/link to build directory)
< ... many more commands ...>

For information about one particular command, you can do something like python setup.py sdist --help.

Testing Your Package

First, you should check that the newly built distribution packages contain the files you expect. On Linux and macOS, you should be able to list the contents of the tar source archive as follows:

$ tar tzf realpython-reader-1.0.0.tar.gz
realpython-reader-1.0.0/
realpython-reader-1.0.0/setup.cfg
realpython-reader-1.0.0/README.md
realpython-reader-1.0.0/reader/
realpython-reader-1.0.0/reader/feed.py
realpython-reader-1.0.0/reader/__init__.py
realpython-reader-1.0.0/reader/viewer.py
realpython-reader-1.0.0/reader/__main__.py
realpython-reader-1.0.0/reader/config.txt
realpython-reader-1.0.0/PKG-INFO
realpython-reader-1.0.0/setup.py
realpython-reader-1.0.0/MANIFEST.in
realpython-reader-1.0.0/realpython_reader.egg-info/
realpython-reader-1.0.0/realpython_reader.egg-info/SOURCES.txt
realpython-reader-1.0.0/realpython_reader.egg-info/requires.txt
realpython-reader-1.0.0/realpython_reader.egg-info/dependency_links.txt
realpython-reader-1.0.0/realpython_reader.egg-info/PKG-INFO
realpython-reader-1.0.0/realpython_reader.egg-info/entry_points.txt
realpython-reader-1.0.0/realpython_reader.egg-info/top_level.txt

On Windows, you can use a utility like 7-zip to look inside the corresponding zip file.

You should see all your source code listed, as well as a few new files that have been created containing information you provided in setup.py. In particular, make sure that all subpackages and supporting files are included.

You can also have a look inside the wheel by unzipping it as if it were a zip file. However, if your source archive contains the files you expect, the wheel should be fine as well.

Newer versions of Twine (1.12.0 and above) can also check that your package description will render properly on PyPI. You can run twine check on the files created in dist:

$ twine check dist/*
Checking distribution dist/realpython_reader-1.0.0-py3-none-any.whl: Passed
Checking distribution dist/realpython-reader-1.0.0.tar.gz: Passed

While it won’t catch all problems you might run into, it will for instance let you know if you are using the wrong content type.

Uploading Your Package

Now you’re ready to actually upload your package to PyPI. For this, you’ll again use the Twine tool, telling it to upload the distribution packages you have built. First, you should upload to TestPyPI to make sure everything works as expected:

$ twine upload --repository-url https://test.pypi.org/legacy/ dist/*

Twine will ask you for your username and password.

Note: If you’ve followed the tutorial using the reader package as an example, the previous command will probably fail with a message saying you are not allowed to upload to the realpython-reader project.

You can change the name in setup.py to something unique, for example test-your-username. Then build the project again and upload the newly built files to TestPyPI.

If the upload succeeds, you can quickly head over to TestPyPI, scroll down, and look at your project being proudly displayed among the new releases! Click on your package and make sure everything looks okay.

If you have been following along using the reader package, the tutorial ends here! While you can play with TestPyPI as much as you want, you shouldn’t upload dummy packages to PyPI just for testing.

However, if you have your own package to publish, then the moment has finally arrived! With all the preparations taken care of, this final step is short:

$ twine upload dist/*

Provide your username and password when requested. That’s it!

Head over to PyPI and look up your package. You can find it either by searching, by looking at the Your projects page, or by going directly to the URL of your project: pypi.org/project/your-package-name/.

Congratulations! Your package is published on PyPI!

`pip install` Your Package

Take a moment to bask in the blue glow of the PyPI web page and (of course) brag to your friends.

Then open up a terminal again. There is one more great pay off!

With your package uploaded to PyPI, you can install it with pip as well:

$ pip install your-package-name

Replace your-package-name with the name you chose for your package. For instance, to install the reader package, you would do the following:

$ pip install realpython-reader

Seeing your own code installed by pip is a wonderful feeling!

Other Useful Tools

Before wrapping up, there are a few other tools that are useful to know about when creating and publishing Python packages.

Virtual Environments

In this guide, we haven’t talked about virtual environments. Virtual environments are very useful when working with different projects, each with their own differing requirements and dependencies.

See the following guides for more information:

In particular, it’s useful to test your package inside a minimal virtual environment to make sure you’re including all necessary dependencies in your setup.py file.

Cookiecutter

One great way to get started with your project is to use Cookiecutter. It sets up your project by asking you a few questions based on a template. Many different templates are available.

First, make sure you have Cookiecutter installed on your system. You can install it from PyPI:

$ pip install cookiecutter

As an example, we’ll use the pypackage-minimal template. To use a template, give Cookiecutter a link to the template:

$ cookiecutter https://github.com/kragniz/cookiecutter-pypackage-minimal
author_name [Louis Taylor]: Real Python
author_email [louis@kragniz.eu]: office@realpython.com
package_name [cookiecutter_pypackage_minimal]: realpython-reader
package_version [0.1.0]:
package_description [...]: Read Real Python tutorials
package_url [...]: https://github.com/realpython/reader
readme_pypi_badge [True]:
readme_travis_badge [True]: False
readme_travis_url [...]:

After you have answered a series of questions, Cookiecutter sets up your project. In this example, the template created the following files and directories:

realpython-reader/
│
├── realpython-reader/
│   └── __init__.py
│
├── tests/
│   ├── __init__.py
│   └── test_sample.py
│
├── README.rst
├── setup.py
└── tox.ini

Cookiecutter’s documentation is extensive and includes a long list of available cookiecutters, as well as tutorials on how to create your own template.

Flit

The history of packaging in Python is quite messy. One common criticism is that using an executable file like setup.py for configuration information is not ideal.

PEP 518 defines an alternative: using a file called pyproject.toml instead. The TOML format is a simple configuration file format:

[…] it is human-usable (unlike JSON), it is flexible enough (unlike configparser), stems from a standard (also unlike configparser), and it is not overly complex (unlike YAML). (Source)

While PEP 518 is already a few years old, the pyproject.toml configuration file is not yet fully supported in the standard tools.

However, there are a few new tools that can publish to PyPI based on pyproject.toml. One such tool is Flit, a great little project that makes it easy to publish simple Python packages. Flit doesn’t support advanced packages like those creating C extensions.

You can pip install flit, and then start using it as follows:

$ flit init
Module name [reader]:
Author []: Real Python
Author email []: office@realpython.com
Home page []: https://github.com/realpython/reader
Choose a license (see http://choosealicense.com/ for more info)
1. MIT - simple and permissive
2. Apache - explicitly grants patent rights
3. GPL - ensures that code based on this is shared with the same terms
4. Skip - choose a license later
Enter 1-4 [1]:

Written pyproject.toml; edit that file to add optional extra info.

The flit init command will create a pyproject.toml file based on the answers you give to a few questions. You might need to edit this file slightly before using it. For the reader project, the pyproject.toml file for Flit ends up looking as follows:

[build-system]
requires = ["flit"]
build-backend = "flit.buildapi"

[tool.flit.metadata]
module = "reader"
dist-name = "realpython-reader"
description-file = "README.md"
author = "Real Python"
author-email = "office@realpython.com"
home-page = "https://github.com/realpython/reader"
classifiers = [
    "License :: OSI Approved :: MIT License",
    "Programming Language :: Python :: 3",
    "Programming Language :: Python :: 3.7",
]
requires-python = ">=3.7"
requires = ["feedparser", "html2text"]

[tool.flit.scripts]
realpython = "reader.__main__:main"

You should recognize most of the items from our original setup.py. One thing to note though is that version and description are missing. This is not a mistake. Flit actually figures these out itself by using __version__ and the docstring defined in the __init__.py file. Flit’s documentation explains everything about the pyproject.toml file.

Flit can build your package and even publish it to PyPI. To build your package, simply do the following:

$ flit build

This creates a source archive and a wheel, exactly like python setup.py sdist bdist_wheel did earlier. To upload your package to PyPI, you can use Twine as earlier. However, you can also use Flit directly:

$ flit publish

The publish command will build your package if necessary, and then upload the files to PyPI, prompting you for your username and password if necessary.

To see Flit in action, have a look at the 2 minute lightning talk from EuroSciPy 2017. The Flit documentation is a great resource for more information. Brett Cannon’s tutorial on packaging up your Python code for PyPI includes a section about Flit.

Poetry

Poetry is another tool that can be used to build and upload your package. It’s quite similar to Flit, especially for the things we’re looking at here.

Before you use Poetry, you need to install it. It’s possible to pip install poetry as well. However, the author recommends that you use a custom installation script to avoid potential dependency conflicts. See the documentation for installation instructions.

With Poetry installed, you start using it with an init command:

$ poetry init

This command will guide you through creating your pyproject.toml config.

Package name [code]: realpython-reader
Version [0.1.0]: 1.0.0
Description []: Read the latest Real Python tutorials
...

This will create a pyproject.toml file based on your answers to questions about your package. Unfortunately, the actual specifications inside the pyproject.toml differ between Flit and Poetry. For Poetry, the pyproject.toml file ends up looking like the following:

[tool.poetry]
name = "realpython-reader"
version = "1.0.0"
description = "Read the latest Real Python tutorials"
readme = "README.md"
homepage = "https://github.com/realpython/reader"
authors = ["Real Python <office@realpython.com>"]
license = "MIT"
packages = [{include = "reader"}]
include = ["reader/*.txt"]

[tool.poetry.dependencies]
python = ">=3.7"
feedparser = ">=5.2"
html2text = ">=2018.1"

[tool.poetry.scripts]
realpython = "reader.__main__:main"

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

Again, you should recognize all these items from the earlier discussion of setup.py. One thing to note is that Poetry will automatically add classifiers based on the license and the version of Python you specify. Poetry also requires you to be explicit about versions of your dependencies. In fact, dependency management is one of the strong points of Poetry.

Just like Flit, Poetry can build and upload packages to PyPI. The build command creates a source archive and a wheel:

$ poetry build

This will create the two usual files in the dist subdirectory, which you can upload using Twine as earlier. You can also use Poetry to publish to PyPI:

$ poetry publish

This will upload your package to PyPI. In addition to building and publishing, Poetry can help you earlier in the process. Similar to Cookiecutter, Poetry can help you start a new project with the new command. It also supports working with virtual environments. See Poetry’s documentation for all the details.

Apart from the slightly different configuration files, Flit and Poetry work very similarly. Poetry is broader in scope as it also aims to help with dependency management, while Flit has been around a little longer. Andrew Pinkham’s article Python’s New Package Landscape covers both Flit and Poetry. Poetry was one of the topics at the special 100th episode of the Python Bytes podcast.

Conclusion

You now know how to prepare your project and upload it to PyPI, so that it can be installed and used by other people. While there are a few steps you need to go through, seeing your own package on PyPI is a great pay off. Having others find your project useful is even better!

In this tutorial, you’ve seen the steps necessary to publish your own package:

Find a good name for your package
Configure your package using setup.py
Build your package
Upload your package to PyPI

In addition, you’ve also seen a few new tools for publishing packages that use the new pyproject.toml configuration file to simplify the process.

If you still have questions, feel free to reach out in the comments section below. Also, the Python Packaging Authority has a lot of information with more detail than we covered here.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python "while" Loops (Indefinite Iteration)

2018-11-07T14:00:00+00:00

Iteration means executing the same block of code over and over, potentially many times. A programming structure that implements iteration is called a loop.

In programming, there are two types of iteration, indefinite and definite:

With indefinite iteration, the number of times the loop is executed isn’t specified explicitly in advance. Rather, the designated block is executed repeatedly as long as some condition is met.
With definite iteration, the number of times the designated block will be executed is specified explicitly at the time the loop starts.

In this tutorial, you’ll:

Learn about the while loop, the Python control structure used for indefinite iteration
See how to break out of a loop or loop iteration prematurely
Explore infinite loops

When you’re finished, you should have a good grasp of how to use indefinite iteration in Python.

Free Bonus: Click here to get our free Python Cheat Sheet that shows you the basics of Python 3, like working with data types, dictionaries, lists, and Python functions.

The `while` Loop

Let’s see how Python’s while statement is used to construct loops. We’ll start simple and embellish as we go.

The format of a rudimentary while loop is shown below:

while <expr>:
    <statement(s)>

<statement(s)> represents the block to be repeatedly executed, often referred to as the body of the loop. This is denoted with indentation, just as in an if statement.

Remember: All control structures in Python use indentation to define blocks. See the discussion on grouping statements in the previous tutorial to review.

The controlling expression, <expr>, typically involves one or more variables that are initialized prior to starting the loop and then modified somewhere in the loop body.

When a while loop is encountered, <expr> is first evaluated in Boolean context. If it is true, the loop body is executed. Then <expr> is checked again, and if still true, the body is executed again. This continues until <expr> becomes false, at which point program execution proceeds to the first statement beyond the loop body.

Consider this loop:

>>>

 1 >>> n = 5
 2 >>> while n > 0:
 3 ...     n -= 1
 4 ...     print(n)
 5 ...
 6 4
 7 3
 8 2
 9 1
10 0

Here’s what’s happening in this example:

n is initially 5. The expression in the while statement header on line 2 is n > 0, which is true, so the loop body executes. Inside the loop body on line 3, n is decremented by 1 to 4, and then printed.
When the body of the loop has finished, program execution returns to the top of the loop at line 2, and the expression is evaluated again. It is still true, so the body executes again, and 3 is printed.
This continues until n becomes 0. At that point, when the expression is tested, it is false, and the loop terminates. Execution would resume at the first statement following the loop body, but there isn’t one in this case.

Note that the controlling expression of the while loop is tested first, before anything else happens. If it’s false to start with, the loop body will never be executed at all:

>>>

>>> n = 0
>>> while n > 0:
...     n -= 1
...     print(n)
...

In the example above, when the loop is encountered, n is 0. The controlling expression n > 0 is already false, so the loop body never executes.

Here’s another while loop involving a list, rather than a numeric comparison:

>>>

>>> a = ['foo', 'bar', 'baz']
>>> while a:
...     print(a.pop(-1))
...
baz
bar
foo

When a list is evaluated in Boolean context, it is truthy if it has elements in it and falsy if it is empty. In this example, a is true as long as it has elements in it. Once all the items have been removed with the .pop() method and the list is empty, a is false, and the loop terminates.

Interruption of Loop Iteration

In each example you have seen so far, the entire body of the while loop is executed on each iteration. Python provides two keywords that terminate a loop iteration prematurely:

break immediately terminates a loop entirely. Program execution proceeds to the first statement following the loop body.
continue immediately terminates the current loop iteration. Execution jumps to the top of the loop, and the controlling expression is re-evaluated to determine whether the loop will execute again or terminate.

The distinction between break and continue is demonstrated in the following diagram:

break and continue

Here’s a script file called break.py that demonstrates the break statement:

 1 n = 5
 2 while n > 0:
 3     n -= 1
 4     if n == 2:
 5         break
 6     print(n)
 7 print('Loop ended.')

Running break.py from a command-line interpreter produces the following output:

C:\Users\john\Documents>python break.py
4
3
Loop ended.

When n becomes 2, the break statement is executed. The loop is terminated completely, and program execution jumps to the print() statement on line 7.

The next script, continue.py, is identical except for a continue statement in place of the break:

 1 n = 5
 2 while n > 0:
 3     n -= 1
 4     if n == 2:
 5         continue
 6     print(n)
 7 print('Loop ended.')

The output of continue.py looks like this:

C:\Users\john\Documents>python continue.py
4
3
1
0
Loop ended.

This time, when n is 2, the continue statement causes termination of that iteration. Thus, 2 isn’t printed. Execution returns to the top of the loop, the condition is re-evaluated, and it is still true. The loop resumes, terminating when n becomes 0, as previously.

The `else` Clause

Python allows an optional else clause at the end of a while loop. This is a unique feature of Python, not found in most other programming languages. The syntax is shown below:

while <expr>:
    <statement(s)>
else:
    <additional_statement(s)>

The <additional_statement(s)> specified in the else clause will be executed when the while loop terminates.

About now, you may be thinking, “How is that useful?” You could accomplish the same thing by putting those statements immediately after the while loop, without the else:

while <expr>:
    <statement(s)>
<additional_statement(s)>

What’s the difference?

In the latter case, without the else clause, <additional_statement(s)> will be executed after the while loop terminates, no matter what.

When <additional_statement(s)> are placed in an else clause, they will be executed only if the loop terminates “by exhaustion”—that is, if the loop iterates until the controlling condition becomes false. If the loop is exited by a break statement, the else clause won’t be executed.

Consider the following example:

>>>

>>> n = 5
>>> while n > 0:
...     n -= 1
...     print(n)
... else:
...     print('Loop done.')
...
4
3
2
1
0
Loop done.

In this case, the loop repeated until the condition was exhausted: n became 0, so n > 0 became false. Because the loop lived out its natural life, so to speak, the else clause was executed. Now observe the difference here:

>>>

>>> n = 5
>>> while n > 0:
...     n -= 1
...     print(n)
...     if n == 2:
...         break
... else:
...     print('Loop done.')
...
4
3
2

This loop is terminated prematurely with break, so the else clause isn’t executed.

It may seem as if the meaning of the word else doesn’t quite fit the while loop as well as it does the if statement. Guido van Rossum, the creator of Python, has actually said that, if he had it to do over again, he’d leave the while loop’s else clause out of the language.

One of the following interpretations might help to make it more intuitive:

Think of the header of the loop (while n > 0) as an if statement (if n > 0) that gets executed over and over, with the else clause finally being executed when the condition becomes false.
Think of else as though it were nobreak, in that the block that follows gets executed if there wasn’t a break.

If you don’t find either of these interpretations helpful, then feel free to ignore them.

When might an else clause on a while loop be useful? One common situation is if you are searching a list for a specific item. You can use break to exit the loop if the item is found, and the else clause can contain code that is meant to be executed if the item isn’t found:

>>>

>>> a = ['foo', 'bar', 'baz', 'qux']
>>> s = 'corge'

>>> i = 0
>>> while i < len(a):
...     if a[i] == s:
...         # Processing for item found
...         break
...     i += 1
... else:
...     # Processing for item not found
...     print(s, 'not found in list.')
...
corge not found in list.

Note: The code shown above is useful to illustrate the concept, but you’d actually be very unlikely to search a list that way.

First of all, lists are usually processed with definite iteration, not a while loop. Definite iteration is covered in the next tutorial in this series.

Secondly, Python provides built-in ways to search for an item in a list. You can use the in operator:

>>>

>>> if s in a:
...     print(s, 'found in list.')
... else:
...     print(s, 'not found in list.')
...
corge not found in list.

The list.index() method would also work. This method raises a ValueError exception if the item isn’t found in the list, so you need to understand exception handling to use it. In Python, you use a try statement to handle an exception. An example is given below:

>>>

>>> try:
...     print(a.index('corge'))
... except ValueError:
...     print(s, 'not found in list.')
...
corge not found in list.

You will learn about exception handling later in this series.

An else clause with a while loop is a bit of an oddity, not often seen. But don’t shy away from it if you find a situation in which you feel it adds clarity to your code!

Infinite Loops

Suppose you write a while loop that theoretically never ends. Sounds weird, right?

Consider this example:

>>>

>>> while True:
...     print('foo')
...
foo
foo
foo
  .
  .
  .
foo
foo
foo
KeyboardInterrupt
Traceback (most recent call last):
  File "<pyshell#2>", line 2, in <module>
    print('foo')

This code was terminated by Ctrl+C, which generates an interrupt from the keyboard. Otherwise, it would have gone on unendingly. Many foo output lines have been removed and replaced by the vertical ellipsis in the output shown.

Clearly, True will never be false, or we’re all in very big trouble. Thus, while True: initiates an infinite loop that will theoretically run forever.

Maybe that doesn’t sound like something you’d want to do, but this pattern is actually quite common. For example, you might write code for a service that starts up and runs forever accepting service requests. “Forever” in this context means until you shut it down, or until the heat death of the universe, whichever comes first.

More prosaically, remember that loops can be broken out of with the break statement. It may be more straightforward to terminate a loop based on conditions recognized within the loop body, rather than on a condition evaluated at the top.

Here’s another variant of the loop shown above that successively removes items from a list using .pop() until it is empty:

>>>

>>> a = ['foo', 'bar', 'baz']
>>> while True:
...     if not a:
...         break
...     print(a.pop(-1))
...
baz
bar
foo

When a becomes empty, not a becomes true, and the break statement exits the loop.

You can also specify multiple break statements in a loop:

while True:
    if <expr1>:  # One condition for loop termination
        break
    ...
    if <expr2>:  # Another termination condition
        break
    ...
    if <expr3>:  # Yet another
        break

In cases like this, where there are multiple reasons to end the loop, it is often cleaner to break out from several different locations, rather than try to specify all the termination conditions in the loop header.

Infinite loops can be very useful. Just remember that you must ensure the loop gets broken out of at some point, so it doesn’t truly become infinite.

Nested `while` Loops

In general, Python control structures can be nested within one another. For example, if/elif/else conditional statements can be nested:

if age < 18:
    if gender == 'M':
        print('son')
    else:
        print('daughter')
elif age >= 18 and age < 65:
    if gender == 'M':
        print('father')
    else:
        print('mother')
else:
    if gender == 'M':
        print('grandfather')
    else:
        print('grandmother')

Similarly, a while loop can be contained within another while loop, as shown here:

>>>

>>> a = ['foo', 'bar']
>>> while len(a):
...     print(a.pop(0))
...     b = ['baz', 'qux']
...     while len(b):
...         print('>', b.pop(0))
...
foo
> baz
> qux
bar
> baz
> qux

A break or continue statement found within nested loops applies to the nearest enclosing loop:

while <expr1>:
    statement
    statement

    while <expr2>:
        statement
        statement
        break  # Applies to while <expr2>: loop

    break  # Applies to while <expr1>: loop

Additionally, while loops can be nested inside if/elif/else statements, and vice versa:

if <expr>:
    statement
    while <expr>:
        statement
        statement
else:
    while <expr>:
        statement
        statement
    statement

while <expr>:
    if <expr>:
        statement
    elif <expr>:
        statement
    else:
        statement

    if <expr>:
        statement

In fact, all the Python control structures can be intermingled with one another to whatever extent you need. That is as it should be. Imagine how frustrating it would be if there were unexpected restrictions like “A while loop can’t be contained within an if statement” or “while loops can only be nested inside one another at most four deep.” You’d have a very difficult time remembering them all.

Seemingly arbitrary numeric or logical limitations are considered a sign of poor program language design. Happily, you won’t find many in Python.

One-Line `while` Loops

As with an if statement, a while loop can be specified on one line. If there are multiple statements in the block that makes up the loop body, they can be separated by semicolons (;):

>>>

>>> n = 5
>>> while n > 0: n -= 1; print(n)

4
3
2
1
0

This only works with simple statements though. You can’t combine two compound statements into one line. Thus, you can specify a while loop all on one line as above, and you write an if statement on one line:

>>>

>>> if True: print('foo')

foo

But you can’t do this:

>>>

>>> while n > 0: n -= 1; if True: print('foo')
SyntaxError: invalid syntax

Remember that PEP 8 discourages multiple statements on one line. So you probably shouldn’t be doing any of this very often anyhow.

Conclusion

In this tutorial, you learned about indefinite iteration using the Python while loop. You’re now able to:

Construct basic and complex while loops
Interrupt loop execution with break and continue
Use the else clause with a while loop
Deal with infinite loops

You should now have a good grasp of how to execute a piece of code repetitively.

The next tutorial in this series covers definite iteration with for loops—recurrent execution where the number of repetitions is specified explicitly.

« Conditional Statements in Python

Python "while" Loops

Python "for" Loops »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Writing Comments in Python (Guide)

2018-11-05T14:00:00+00:00

When writing code in Python, it’s important to make sure that your code can be easily understood by others. Giving variables obvious names, defining explicit functions, and organizing your code are all great ways to do this.

Another awesome and easy way to increase the readability of your code is by using comments!

In this tutorial, you’ll cover some of the basics of writing comments in Python. You’ll learn how to write comments that are clean and concise, and when you might not need to write any comments at all.

You’ll also learn:

Why it’s so important to comment your code
Best practices for writing comments in Python
Types of comments you might want to avoid
How to practice writing cleaner comments

Free Bonus: 5 Thoughts On Python Mastery, a free course for Python developers that shows you the roadmap and the mindset you'll need to take your Python skills to the next level.

Why Commenting Your Code Is So Important

Comments are an integral part of any program. They can come in the form of module-level docstrings, or even inline explanations that help shed light on a complex function.

Before diving into the different types of comments, let’s take a closer look at why commenting your code is so important.

Consider the following two scenarios in which a programmer decided not to comment their code.

When Reading Your Own Code

Client A wants a last-minute deployment for their web service. You’re already on a tight deadline, so you decide to just make it work. All that “extra” stuff—documentation, proper commenting, and so forth—you’ll add that later.

The deadline comes, and you deploy the service, right on time. Whew!

You make a mental note to go back and update the comments, but before you can put it on your to-do list, your boss comes over with a new project that you need to get started on immediately. Within a few days, you’ve completely forgotten that you were supposed to go back and properly comment the code you wrote for Client A.

Fast forward six months, and Client A needs a patch built for that same service to comply with some new requirements. It’s your job to maintain it, since you were the one who built it in the first place. You open up your text editor and…

What did you even write?!

You spend hours parsing through your old code, but you’re completely lost in the mess. You were in such a rush at the time that you didn’t name your variables properly or even set your functions up in the proper control flow. Worst of all, you don’t have any comments in the script to tell you what’s what!

Developers forget what their own code does all the time, especially if it was written a long time ago or under a lot of pressure. When a deadline is fast approaching, and hours in front of the computer have led to bloodshot eyes and cramped hands, that pressure can be reflected in the form of code that is messier than usual.

Once the project is submitted, many developers are simply too tired to go back and comment their code. When it’s time to revisit it later down the line, they can spend hours trying to parse through what they wrote.

Writing comments as you go is a great way to prevent the above scenario from happening. Be nice to Future You!

When Others Are Reading Your Code

Imagine you’re the only developer working on a small Django project. You understand your own code pretty well, so you don’t tend to use comments or any other sort of documentation, and you like it that way. Comments take time to write and maintain, and you just don’t see the point.

The only problem is, by the end of the year your “small Django project” has turned into a “20,000 lines of code” project, and your supervisor is bringing on additional developers to help maintain it.

The new devs work hard to quickly get up to speed, but within the first few days of working together, you’ve realized that they’re having some trouble. You used some quirky variable names and wrote with super terse syntax. The new hires spend a lot of time stepping through your code line by line, trying to figure out how it all works. It takes a few days before they can even help you maintain it!

Using comments throughout your code can help other developers in situations like this one. Comments help other devs skim through your code and gain an understanding of how it all works very quickly. You can help ensure a smooth transition by choosing to comment your code from the outset of a project.

How to Write Comments in Python

Now that you understand why it’s so important to comment your code, let’s go over some basics so you know how to do it properly.

Python Commenting Basics

Comments are for developers. They describe parts of the code where necessary to facilitate the understanding of programmers, including yourself.

To write a comment in Python, simply put the hash mark # before your desired comment:

# This is a comment

Python ignores everything after the hash mark and up to the end of the line. You can insert them anywhere in your code, even inline with other code:

print("This will run.")  # This won't run

When you run the above code, you will only see the output This will run. Everything else is ignored.

Comments should be short, sweet, and to the point. While PEP 8 advises keeping code at 79 characters or fewer per line, it suggests a max of 72 characters for inline comments and docstrings. If your comment is approaching or exceeding that length, then you’ll want to spread it out over multiple lines.

Python Multiline Comments

Unfortunately, Python doesn’t have a way to write multiline comments as you can in languages such as C, Java, and Go:

# So you can't
just do this
in python

In the above example, the first line will be ignored by the program, but the other lines will raise a Syntax Error.

In contrast, a language like Java will allow you to spread a comment out over multiple lines quite easily:

/* You can easily
write multiline
comments in Java */

Everything between /* and */ is ignored by the program.

While Python doesn’t have native multiline commenting functionality, you can create multiline comments in Python. There are two simple ways to do so.

The first way is simply by pressing the return key after each line, adding a new hash mark and continuing your comment from there:

def multiline_example():
    # This is a pretty good example
    # of how you can spread comments
    # over multiple lines in Python

Each line that starts with a hash mark will be ignored by the program.

Another thing you can do is use multiline strings by wrapping your comment inside a set of triple quotes:

"""
If I really hate pressing `enter` and
typing all those hash marks, I could
just do this instead
"""

This is like multiline comments in Java, where everything enclosed in the triple quotes will function as a comment.

While this gives you the multiline functionality, this isn’t technically a comment. It’s a string that’s not assigned to any variable, so it’s not called or referenced by your program. Still, since it’ll be ignored at runtime and won’t appear in the bytecode, it can effectively act as a comment. (You can take a look at this article for proof that these strings won’t show up in the bytecode.)

However, be careful where you place these multiline “comments.” Depending on where they sit in your program, they could turn into docstrings, which are pieces of documentation that are associated with a function or method. If you slip one of these bad boys right after a function definition, then what you intended to be a comment will become associated with that object.

Be careful where you use these, and when in doubt, just put a hash mark on each subsequent line. If you’re interested in learning more about docstrings and how to associate them with modules, classes, and the like, check out our tutorial on Documenting Python Code.

Python Commenting Shortcuts

It can be tedious to type out all those hash marks every time you need to add a comment. So what can you do to speed things up a bit? Here are a few tricks to help you out when commenting.

One of the first things you can do is use multiple cursors. That’s exactly what it sounds like: placing more than one cursor on your screen to accomplish a task. Simply hold down the Ctrl or Cmd key while you left-click, and you should see the blinking lines on your screen:

This is most effective when you need to comment the same thing in several places.

What if you’ve got a long stretch of text that needs to be commented out? Say you don’t want a defined function to run in order to check for a bug. Clicking each and every line to comment it out could take a lot of time! In these cases, you’ll want to toggle comments instead. Simply select the desired code and press Ctrl+/ on PC, or Cmd+/ on Mac:

All the highlighted text will be prepended with a hash mark and ignored by the program.

If your comments are getting too unwieldy, or the comments in a script you’re reading are really long, then your text editor may give you the option to collapse them using the small down arrow on the left-hand side:

Simply click the arrow to hide the comments. This works best with long comments spread out over multiple lines, or docstrings that take up most of the start of a program.

Combining these tips will make commenting your code quick, easy, and painless!

Python Commenting Best Practices

While it’s good to know how to write comments in Python, it’s just as vital to make sure that your comments are readable and easy to understand.

Take a look at these tips to help you write comments that really support your code.

When Writing Code for Yourself

You can make life easier for yourself by commenting your own code properly. Even if no one else will ever see it, you’ll see it, and that’s enough reason to make it right. You’re a developer after all, so your code should be easy for you to understand as well.

One extremely useful way to use comments for yourself is as an outline for your code. If you’re not sure how your program is going to turn out, then you can use comments as a way to keep track of what’s left to do, or even as a way of tracking the high-level flow of your program. For instance, use comments to outline a function in pseudo-code:

from collections import defaultdict

def get_top_cities(prices):
    top_cities = defaultdict(int)

    # For each price range
        # Get city searches in that price
        # Count num times city was searched
        # Take top 3 cities & add to dict

    return dict(top_cities)

These comments plan out get_top_cities(). Once you know exactly what you want your function to do, you can work on translating that to code.

Using comments like this can help keep everything straight in your head. As you walk through your program, you’ll know what’s left to do in order to have a fully functional script. After “translating” the comments to code, remember to remove any comments that have become redundant so that your code stays crisp and clean.

You can also use comments as part of the debugging process. Comment out the old code and see how that affects your output. If you agree with the change, then don’t leave the code commented out in your program, as it decreases readability. Delete it and use version control if you need to bring it back.

Finally, use comments to define tricky parts of your own code. If you put a project down and come back to it months or years later, you’ll spend a lot of time trying to get reacquainted with what you wrote. In case you forget what your own code does, do Future You a favor and mark it down so that it will be easier to get back up to speed later on.

When Writing Code for Others

People like to skim and jump back and forth through text, and reading code is no different. The only time you’ll probably read through code line by line is when it isn’t working and you have to figure out what’s going on.

In most other cases, you’ll take a quick glance at variables and function definitions in order to get the gist. Having comments to explain what’s happening in plain English can really assist a developer in this position.

Be nice to your fellow devs and use comments to help them skim through your code. Inline comments should be used sparingly to clear up bits of code that aren’t obvious on their own. (Of course, your first priority should be to make your code stand on its own, but inline comments can be useful in this regard.)

If you have a complicated method or function whose name isn’t easily understandable, you may want to include a short comment after the def line to shed some light:

def complicated_function(s):
    # This function does something complicated

This can help other devs who are skimming your code get a feel for what the function does.

For any public functions, you’ll want to include an associated docstring, whether it’s complicated or not:

def sparsity_ratio(x: np.array) -> float:
    """Return a float

    Percentage of values in array that are zero or NaN
    """

This string will become the .__doc__ attribute of your function and will officially be associated with that specific method. The PEP 257 docstring guidelines will help you to structure your docstring. These are a set of conventions that developers generally use when structuring docstrings.

The PEP 257 guidelines have conventions for multiline docstrings as well. These docstrings appear right at the top of a file and include a high-level overview of the entire script and what it’s supposed to do:

# -*- coding: utf-8 -*-
"""A module-level docstring

Notice the comment above the docstring specifying the encoding.
Docstrings do appear in the bytecode, so you can access this through
the ``__doc__`` attribute. This is also what you'll see if you call
help() on a module or any other Python object.
"""

A module-level docstring like this one will contain any pertinent or need-to-know information for the developer reading it. When writing one, it’s recommended to list out all classes, exceptions, and functions as well as a one-line summary for each.

Python Commenting Worst Practices

Just as there are standards for writing Python comments, there are a few types of comments that don’t lead to Pythonic code. Here are just a few.

Avoid: W.E.T. Comments

Your comments should be D.R.Y. The acronym stands for the programming maxim “Don’t Repeat Yourself.” This means that your code should have little to no redundancy. You don’t need to comment a piece of code that sufficiently explains itself, like this one:

return a  # Returns a

We can clearly see that a is returned, so there’s no need to explicitly state this in a comment. This makes comments W.E.T., meaning you “wrote everything twice.” (Or, for the more cynical out there, “wasted everyone’s time.”)

W.E.T. comments can be a simple mistake, especially if you used comments to plan out your code before writing it. But once you’ve got the code running well, be sure to go back and remove comments that have become unnecessary.

Avoid: Smelly Comments

Comments can be a sign of “code smell,” which is anything that indicates there might be a deeper problem with your code. Code smells try to mask the underlying issues of a program, and comments are one way to try and hide those problems. Comments should support your code, not try to explain it away. If your code is poorly written, no amount of commenting is going to fix it.

Let’s take this simple example:

# A dictionary of families who live in each city
mydict = {
    "Midtown": ["Powell", "Brantley", "Young"],
    "Norcross": ["Montgomery"], 
    "Ackworth": []
}

def a(dict):
    # For each city
    for p in dict:
        # If there are no families in the city
        if not mydict[p]:
            # Say that there are no families
            print("None.")

This code is quite unruly. There’s a comment before every line explaining what the code does. This script could have been made simpler by assigning obvious names to variables, functions, and collections, like so:

families_by_city = {
    "Midtown": ["Powell", "Brantley", "Young"],
    "Norcross": ["Montgomery"],
    "Ackworth": [],
}

def no_families(cities):
    for city in cities:
        if not families_by_city[city]:
            print(f"No families in {city}.")

By using obvious naming conventions, we were able to remove all unnecessary comments and reduce the length of the code as well!

Your comments should rarely be longer than the code they support. If you’re spending too much time explaining what you did, then you need to go back and refactor to make your code more clear and concise.

Avoid: Rude Comments

This is something that’s likely to come up when working on a development team. When several people are all working on the same code, others are going to be going in and reviewing what you’ve written and making changes. From time to time, you might come across someone who dared to write a comment like this one:

# Put this here to fix Ryan's stupid-a** mistake

Honestly, it’s just a good idea to not do this. It’s not okay if it’s your friend’s code, and you’re sure they won’t be offended by it. You never know what might get shipped to production, and how is it going to look if you’d accidentally left that comment in there, and a client discovered it down the road? You’re a professional, and including vulgar words in your comments is not the way to show that.

How to Practice Commenting

The simplest way to start writing more Pythonic comments is just to do it!

Start writing comments for yourself in your own code. Make it a point to include simple comments from now on where necessary. Add some clarity to complex functions, and put a docstring at the top of all your scripts.

Another good way to practice is to go back and review old code that you’ve written. See where anything might not make sense, and clean up the code. If it still needs some extra support, add a quick comment to help clarify the code’s purpose.

This is an especially good idea if your code is up on GitHub and people are forking your repo. Help them get started by guiding them through what you’ve already done.

You can also give back to the community by commenting other people’s code. If you’ve downloaded something from GitHub and had trouble sifting through it, add comments as you come to understand what each piece of code does.

“Sign” your comment with your initials and the date, and then submit your changes as a pull request. If your changes are merged, you could be helping dozens if not hundreds of developers like yourself get a leg up on their next project.

Conclusion

Learning to comment well is a valuable tool. Not only will you learn how to write more clearly and concisely in general, but you’ll no doubt gain a deeper understanding of Python as well.

Knowing how to write comments in Python can make life easier for all developers, including yourself! They can help other devs get up to speed on what your code does, and help you get re-acquainted with old code of your own.

By noticing when you’re using comments to try and support poorly written code, you’ll be able to go back and modify your code to be more robust. Commenting previously written code, whether your own or another developer’s, is a great way to practice writing clean comments in Python.

As you learn more about documenting your code, you can consider moving on to the next level of documentation. Check out our tutorial on Documenting Python Code to take the next step.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Setting Up Python for Machine Learning on Windows

2018-10-31T14:00:00+00:00

Python has been largely used for numerical and scientific applications in the last years. However, to perform numerical computations in an efficient manner, Python relies on external libraries, sometimes implemented in other languages, such as the NumPy library, which is partly implemented using the Fortran language.

Due to these dependencies, sometimes it isn’t trivial to set up an environment for numerical computations, linking all the necessary libraries. It’s common for people to struggle to get things working in workshops involving the use of Python for machine learning, especially when they are using an operating system that lacks a package management system, such as Windows.

In this article, you’ll:

Walk through the details for setting up a Python environment for numerical computations on a Windows operating system
Be introduced to Anaconda, a Python distribution proposed to circumvent these setup problems
See how to install the distribution on a Windows machine and use its tools to manage packages and environments
Use the installed Python stack to build a neural network and train it to solve a classic classification problem

Free Bonus: Click here to get access to a Conda cheat sheet with handy usage examples for managing your Python environment and packages.

Introducing Anaconda and Conda

Since 2011, Python has included pip, a package management system used to install and manage software packages written in Python. However, for numerical computations, there are several dependencies that are not written in Python, so the initial releases of pip could not solve the problem by themselves.

To circumvent this problem, Continuum Analytics released Anaconda, a Python distribution focused on scientific applications and Conda, a package and environment management system, which is used by the Anaconda distribution. It’s worth noticing that the more recent versions of pip can handle external dependencies using wheels, but, by using Anaconda, you’ll be able to install critical libraries for data science more smoothly. (You can read more on this discussion here.)

Although Conda is tightly coupled to the Anaconda Python Distribution, the two are distinct projects with different goals:

Anaconda is a full distribution of the software in the PyData ecosystem, including Python itself along with binaries for several third-party open-source projects. Besides Anaconda, there’s also Miniconda, which is a minimal Python distribution including basically Conda and its dependencies so that you can install only the packages you need, from scratch
Conda is a package, dependency, and environment management system that could be installed without the Anaconda or Miniconda distribution. It runs on Windows, macOS, and Linux and was created for Python programs, but it can package and distribute software for any language. The main purpose is to solve external dependencies issues in an easy way, by downloading pre-compiled versions of software.

In this sense, it is more like a cross-platform version of a general purpose package manager such as APT or YUM, which helps to find and install packages in a language-agnostic way. Also, Conda is an environment manager, so if you need a package that requires a different version of Python, by using Conda, it is possible to set up a separate environment with a totally different version of Python, maintaining your usual version of Python on your default environment.

There’s a lot of discussion regarding the creation of another package management system for the Python ecosystem. It’s worth mentioning that Conda’s creators pushed Python standard packaging to the limit and only created a second tool when it was clear that it was the only reasonable way forward.

Curiously, even Guido van Rossum, at his speech at the inaugural PyData meetup in 2012, said that, when it comes to packaging, “it really sounds like your needs are so unusual compared to the larger Python community that you’re just better off building your own.” (You can watch a video of this discussion.) More information about this discussion can be found here and here.

Anaconda and Miniconda have become the most popular Python distributions, widely used for data science and machine learning in various companies and research laboratories. They are free and open source projects and currently include 1400+ packages in the repository. In the following section, we’ll go through the installation of the Miniconda Python distribution on a Windows machine.

Installing the Miniconda Python Distribution

In this section, you’ll see step-by-step how to set up a data science Python environment on Windows. Instead of the full Anaconda distribution, you’ll be using Miniconda to set up a minimal environment containing only Conda and its dependencies, and you’ll use that to install the necessary packages.

The installation processes for Miniconda and Anaconda are very similar. The basic difference is that Anaconda provides an environment with a lot of pre-installed packages, many of which are never used. (You can check the list here.) Miniconda is minimalist and clean, and it allows you to easily install any of Anaconda’s packages.

In this article, the focus will be on using the command line interface (CLI) to set up the packages and environments. However, it’s possible to use Conda to install Anaconda Navigator, a graphical user interface (GUI), if you wish.

Miniconda can be installed using an installer available here. You’ll notice there are installers for Windows, macOS, and Linux, and for 32-bit or 64-bit operating systems. You should consider the appropriate architecture according to your Windows installation and download the Python 3.x version (at the time of writing this article, 3.7).

There’s no reason to use Python 2 on a fresh project anymore, and if you do need Python 2 on some project you’re working on, due to some library that has not been updated, it is possible to set up a Python 2 environment using Conda, even if you installed the Miniconda Python 3.x distribution, as you will see in the next section.

After the download finishes, you just have to run the installer and follow the installation steps:

Click on Next on the welcome screen:

Click on I Agree to agree to the license terms:

Choose the installation type and click Next. Another advantage of using Anaconda or Miniconda is that it is possible to install the distribution using a local account. (It isn’t necessary to have an administrator account.) If this is the case, choose Just Me. Otherwise, if you have an administrator account, you may choose All Users:

Choose the install location and click Next. If you’ve chosen to install just for you, the default location will be the folder Miniconda3 under your user’s personal folder. It’s important not to use spaces in the folder names in the path to Miniconda, since many Python packages have problems when spaces are used in folder names:

In Advanced Installation Options, the suggestion is to use the default choices, which are to not add Anaconda to the PATH environment variable and to register Anaconda as the default Python. Click Install to begin installation:

Wait while the installer copies the files:

When the installation completes, click on Next:

Click on Finish to finish the installation and close the installer:

As Anaconda was not included in the PATH environment variable, its commands won’t work in the Windows default command prompt. To use the distribution, you should start its own command prompt, which can be done by clicking on the Start button and on Anaconda Prompt under Anaconda3 (64 bit):

When the prompt opens, you can check if Conda is available by running conda --version:

(base) C:\Users\IEUser>conda --version
conda 4.5.11

To get more information about the installation, you can run conda info:

(base) C:\Users\IEUser>conda info

     active environment : base
    active env location : C:\Users\IEUser\Miniconda3
            shell level : 1
       user config file : C:\Users\IEUser\.condarc
 populated config files : C:\Users\IEUser\.condarc
          conda version : 4.5.11
    conda-build version : not installed
         python version : 3.7.0.final.0
       base environment : C:\Users\IEUser\Miniconda3  (writable)
           channel URLs : https://repo.anaconda.com/pkgs/main/win-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/free/win-64
                          https://repo.anaconda.com/pkgs/free/noarch
                          https://repo.anaconda.com/pkgs/r/win-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/pro/win-64
                          https://repo.anaconda.com/pkgs/pro/noarch
                          https://repo.anaconda.com/pkgs/msys2/win-64
                          https://repo.anaconda.com/pkgs/msys2/noarch
          package cache : C:\Users\IEUser\Miniconda3\pkgs
                          C:\Users\IEUser\AppData\Local\conda\conda\pkgs
       envs directories : C:\Users\IEUser\Miniconda3\envs
                          C:\Users\IEUser\AppData\Local\conda\conda\envs
                          C:\Users\IEUser\.conda\envs
               platform : win-64
             user-agent : conda/4.5.11 requests/2.19.1 CPython/3.7.0 Windows/10 Windows/10.0.17134
          administrator : False
             netrc file : None
           offline mode : False

Now that you have Miniconda installed, let’s see how Conda environments work.

Understanding Conda Environments

When you start developing a project from scratch, it’s recommended that you use the latest versions of the libraries you need. However, when working with someone else’s project, such as when running an example from Kaggle or Github, you may need to install specific versions of packages or even another version of Python due to compatibility issues.

This problem may also occur when you try to run an application you’ve developed long ago, which uses a particular library version that does not work with your application anymore due to updates.

Virtual environments are a solution to this kind of problem. By using them, it is possible to create multiple environments, each one with different versions of packages. A typical Python set up includes Virtualenv, a tool to create isolated Python virtual environments, widely used in the Python community.

Conda includes its own environment manager and presents some advantages over Virtualenv, especially concerning numerical applications, such as the ability to manage non-Python dependencies and the ability to manage different versions of Python, which is not possible with Virtualenv. Besides that, Conda environments are entirely compatible with default Python packages that may be installed using pip.

Miniconda installation provides Conda and a root environment with a version of Python and some basic packages installed. Besides this root environment, it is possible to set up additional environments including different versions of Python and packages.

Using the Anaconda prompt, it is possible to check the available Conda environments by running conda env list:

(base) C:\Users\IEUser>conda env list
# conda environments:
#
base                  *  C:\Users\IEUser\Miniconda3

This base environment is the root environment, created by the Miniconda installer. It is possible to create another environment, named otherenv, by running conda create --name otherenv:

(base) C:\Users\IEUser>conda create --name otherenv
Solving environment: done

## Package Plan ##

  environment location: C:\Users\IEUser\Miniconda3\envs\otherenv


Proceed ([y]/n)? y

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate otherenv
#
# To deactivate an active environment, use
#
#     $ conda deactivate

As notified after the environment creation process is finished, it is possible to activate the otherenv environment by running conda activate otherenv. You’ll notice the environment has changed by the indication between parentheses in the beginning of the prompt:

(base) C:\Users\IEUser>conda activate otherenv

(otherenv) C:\Users\IEUser>

You can open the Python interpreter within this environment by running python:

(otherenv) C:\Users\IEUser>python
Python 3.7.0 (default, Jun 28 2018, 08:04:48) [MSC v.1912 64 bit (AMD64)] :: Anaconda, Inc. on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>

The environment includes Python 3.7.0, the same version included in the root base environment. To exit the Python interpreter, just run quit():

>>> quit()

(otherenv) C:\Users\IEUser>

To deactivate the otherenv environment and go back to the root base environment, you should run deactivate:

(otherenv) C:\Users\IEUser>deactivate

(base) C:\Users\IEUser>

As mentioned earlier, Conda allows you to easily create environments with different versions of Python, which is not straightforward with Virtualenv. To include a different Python version within an environment, you have to specify it by using python=<version> when running conda create. For example, to create an environment named py2 with Python 2.7, you have to run conda create --name py2 python=2.7:

(base) C:\Users\IEUser>conda create --name py2 python=2.7
Solving environment: done

## Package Plan ##

  environment location: C:\Users\IEUser\Miniconda3\envs\py2

  added / updated specs:
    - python=2.7


The following NEW packages will be INSTALLED:

    certifi:        2018.8.24-py27_1
    pip:            10.0.1-py27_0
    python:         2.7.15-he216670_0
    setuptools:     40.2.0-py27_0
    vc:             9-h7299396_1
    vs2008_runtime: 9.00.30729.1-hfaea7d5_1
    wheel:          0.31.1-py27_0
    wincertstore:   0.2-py27hf04cefb_0

Proceed ([y]/n)? y

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate py2
#
# To deactivate an active environment, use
#
#     $ conda deactivate


(base) C:\Users\IEUser>

As shown by the output of conda create, this time some new packages were installed, since the new environment uses Python 2. You can check the new environment indeed uses Python 2 by activating it and running the Python interpreter:

(base) C:\Users\IEUser>conda activate py2

(py2) C:\Users\IEUser>python
Python 2.7.15 |Anaconda, Inc.| (default, May  1 2018, 18:37:09) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>

Now, if you run conda env list, you should see the two environments that were created, besides the root base environment:

(py2) C:\Users\IEUser>conda env list
# conda environments:
#
base                     C:\Users\IEUser\Miniconda3
otherenv                 C:\Users\IEUser\Miniconda3\envs\otherenv
py2               *  C:\Users\IEUser\Miniconda3\envs\py2


(py2) C:\Users\IEUser>

In the list, the asterisk indicates the activated environment. It is possible to remove an environment by running conda remove --name <environment name> --all. Since it is not possible to remove an activated environment, you should first deactivate the py2 environment, to remove it:

(py2) C:\Users\IEUser>deactivate

(base) C:\Users\IEUser>conda remove --name py2 --all

Remove all packages in environment C:\Users\IEUser\Miniconda3\envs\py2:


## Package Plan ##

  environment location: C:\Users\IEUser\Miniconda3\envs\py2


The following packages will be REMOVED:

    certifi:        2018.8.24-py27_1
    pip:            10.0.1-py27_0
    python:         2.7.15-he216670_0
    setuptools:     40.2.0-py27_0
    vc:             9-h7299396_1
    vs2008_runtime: 9.00.30729.1-hfaea7d5_1
    wheel:          0.31.1-py27_0
    wincertstore:   0.2-py27hf04cefb_0

Proceed ([y]/n)? y


(base) C:\Users\IEUser>

Now that you’ve covered the basics of managing environments with Conda, let’s see how to manage packages within the environments.

Understanding Basic Package Management With Conda

Within each environment, packages of software can be installed using the Conda package manager. The root base environment created by the Miniconda installer includes some packages by default that are not part of Python standard library.

The default installation includes the minimum packages necessary to use Conda. To check the list of installed packages in an environment, you just have to make sure it is activated and run conda list. In the root environment, the following packages are installed by default:

(base) C:\Users\IEUser>conda list
# packages in environment at C:\Users\IEUser\Miniconda3:
#
# Name                    Version                   Build  Channel
asn1crypto                0.24.0                   py37_0
ca-certificates           2018.03.07                    0
certifi                   2018.8.24                py37_1
cffi                      1.11.5           py37h74b6da3_1
chardet                   3.0.4                    py37_1
conda                     4.5.11                   py37_0
conda-env                 2.6.0                         1
console_shortcut          0.1.1                         3
cryptography              2.3.1            py37h74b6da3_0
idna                      2.7                      py37_0
menuinst                  1.4.14           py37hfa6e2cd_0
openssl                   1.0.2p               hfa6e2cd_0
pip                       10.0.1                   py37_0
pycosat                   0.6.3            py37hfa6e2cd_0
pycparser                 2.18                     py37_1
pyopenssl                 18.0.0                   py37_0
pysocks                   1.6.8                    py37_0
python                    3.7.0                hea74fb7_0
pywin32                   223              py37hfa6e2cd_1
requests                  2.19.1                   py37_0
ruamel_yaml               0.15.46          py37hfa6e2cd_0
setuptools                40.2.0                   py37_0
six                       1.11.0                   py37_1
urllib3                   1.23                     py37_0
vc                        14                   h0510ff6_3
vs2015_runtime            14.0.25123                    3
wheel                     0.31.1                   py37_0
win_inet_pton             1.0.1                    py37_1
wincertstore              0.2                      py37_0
yaml                      0.1.7                hc54c509_2

(base) C:\Users\IEUser>

To manage the packages, you should also use Conda. Next, let’s see how to search, install, update, and remove packages using Conda.

Searching and Installing Packages

Packages are installed from repositories called channels by Conda, and some default channels are configured by the installer. To search for a specific package, you can run conda search <package name>. For example, this is how you search for the keras package (a machine learning library):

(base) C:\Users\IEUser>conda search keras
Loading channels: done
# Name                  Version           Build  Channel
keras                     2.0.8  py35h15001cb_0  pkgs/main
keras                     2.0.8  py36h65e7a35_0  pkgs/main
keras                     2.1.2          py35_0  pkgs/main
keras                     2.1.2          py36_0  pkgs/main
keras                     2.1.3          py35_0  pkgs/main
keras                     2.1.3          py36_0  pkgs/main

... (more)

According to the previous output, there are different versions of the package and different builds for each version, such as for Python 3.5 and 3.6.

The previous search shows only exact matches for packages named keras. To perform a broader search, including all packages containing keras in their names, you should use the wildcard *. For example, when you run conda search *keras*, you get the following:

(base) C:\Users\IEUser>conda search *keras*
Loading channels: done
# Name                  Version           Build  Channel
keras                     2.0.8  py35h15001cb_0  pkgs/main
keras                     2.0.8  py36h65e7a35_0  pkgs/main
keras                     2.1.2          py35_0  pkgs/main
keras                     2.1.2          py36_0  pkgs/main
keras                     2.1.3          py35_0  pkgs/main
keras                     2.1.3          py36_0  pkgs/main

... (more)

keras-applications           1.0.2          py35_0  pkgs/main
keras-applications           1.0.2          py36_0  pkgs/main
keras-applications           1.0.4          py35_0  pkgs/main

... (more)

keras-base                2.2.0          py35_0  pkgs/main
keras-base                2.2.0          py36_0  pkgs/main

... (more)

As the previous output shows, there are some other keras related packages in the default channels.

To install a package, you should run conda install <package name>. By default, the newest version of the package will be installed in the active environment. So, let’s install the package keras in the environment otherenv that you’ve already created:

(base) C:\Users\IEUser>conda activate otherenv

(otherenv) C:\Users\IEUser>conda install keras
Solving environment: done

## Package Plan ##

  environment location: C:\Users\IEUser\Miniconda3\envs\otherenv

  added / updated specs:
    - keras


The following NEW packages will be INSTALLED:

    _tflow_1100_select:  0.0.3-mkl
    absl-py:             0.4.1-py36_0
    astor:               0.7.1-py36_0
    blas:                1.0-mkl
    certifi:             2018.8.24-py36_1
    gast:                0.2.0-py36_0
    grpcio:              1.12.1-py36h1a1b453_0
    h5py:                2.8.0-py36h3bdd7fb_2
    hdf5:                1.10.2-hac2f561_1
    icc_rt:              2017.0.4-h97af966_0
    intel-openmp:        2018.0.3-0
    keras:               2.2.2-0
    keras-applications:  1.0.4-py36_1
    keras-base:          2.2.2-py36_0
    keras-preprocessing: 1.0.2-py36_1
    libmklml:            2018.0.3-1
    libprotobuf:         3.6.0-h1a1b453_0
    markdown:            2.6.11-py36_0
    mkl:                 2019.0-117
    mkl_fft:             1.0.4-py36h1e22a9b_1
    mkl_random:          1.0.1-py36h77b88f5_1
    numpy:               1.15.1-py36ha559c80_0
    numpy-base:          1.15.1-py36h8128ebf_0
    pip:                 10.0.1-py36_0
    protobuf:            3.6.0-py36he025d50_0
    python:              3.6.6-hea74fb7_0
    pyyaml:              3.13-py36hfa6e2cd_0
    scipy:               1.1.0-py36h4f6bf74_1
    setuptools:          40.2.0-py36_0
    six:                 1.11.0-py36_1
    tensorboard:         1.10.0-py36he025d50_0
    tensorflow:          1.10.0-mkl_py36hb361250_0
    tensorflow-base:     1.10.0-mkl_py36h81393da_0
    termcolor:           1.1.0-py36_1
    vc:                  14-h0510ff6_3
    vs2013_runtime:      12.0.21005-1
    vs2015_runtime:      14.0.25123-3
    werkzeug:            0.14.1-py36_0
    wheel:               0.31.1-py36_0
    wincertstore:        0.2-py36h7fe50ca_0
    yaml:                0.1.7-hc54c509_2
    zlib:                1.2.11-h8395fce_2

Proceed ([y]/n)?

Conda manages the necessary dependencies for a package when it is installed. Since the package keras has a lot of dependencies, when you install it, Conda manages to install this big list of packages.

It’s worth noticing that, since the keras package’s newest build uses Python 3.6 and the otherenv environment was created using Python 3.7, the package python version 3.6.6 was included as a dependency. After confirming the installation, you can check that the Python version for the otherenv environment is downgraded to the 3.6.6 version.

Sometimes, you don’t want packages to be downgraded, and it would be better to just create a new environment with the necessary version of Python. To check the list of new packages, updates, and downgrades necessary for a package without installing it, you should use the parameter --dry-run. For example, to check the packages that will be changed by the installation of the package keras, you should run the following:

(otherenv) C:\Users\IEUser>conda install keras --dry-run

However, if necessary, it is possible to change the default Python of a Conda environment by installing a specific version of the package python. To demonstrate that, let’s create a new environment called envpython:

(otherenv) C:\Users\IEUser>conda create --name envpython
Solving environment: done

## Package Plan ##

  environment location: C:\Users\IEUser\Miniconda3\envs\envpython


Proceed ([y]/n)? y

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate envpython
#
# To deactivate an active environment, use
#
#     $ conda deactivate

As you saw before, since the root base environment uses Python 3.7, envpython is created including this same version of Python:

(base) C:\Users\IEUser>conda activate envpython

(envpython) C:\Users\IEUser>python
Python 3.7.0 (default, Jun 28 2018, 08:04:48) [MSC v.1912 64 bit (AMD64)] :: Anaconda, Inc. on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> quit()

(envpython) C:\Users\IEUser>

To install a specific version of a package, you can run conda install <package name>=<version>. For example, this is how you install Python 3.6 in the envpython environment:

(envpython) C:\Users\IEUser>conda install python=3.6
Solving environment: done

## Package Plan ##

  environment location: C:\Users\IEUser\Miniconda3\envs\envpython

  added / updated specs:
    - python=3.6


The following NEW packages will be INSTALLED:

    certifi:        2018.8.24-py36_1
    pip:            10.0.1-py36_0
    python:         3.6.6-hea74fb7_0
    setuptools:     40.2.0-py36_0
    vc:             14-h0510ff6_3
    vs2015_runtime: 14.0.25123-3
    wheel:          0.31.1-py36_0
    wincertstore:   0.2-py36h7fe50ca_0

Proceed ([y]/n)?

In case you need to install more than one package in an environment, it is possible to run conda install only once, passing the names of the packages. To illustrate that, let’s install numpy, scipy, and matplotlib, basic packages for numerical computation in the root base environment:

(envpython) C:\Users\IEUser>deactivate

(base) C:\Users\IEUser>conda install numpy scipy matplotlib
Solving environment: done

## Package Plan ##

  environment location: C:\Users\IEUser\Miniconda3

  added / updated specs:
    - matplotlib
    - numpy
    - scipy


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    libpng-1.6.34              |       h79bbb47_0         1.3 MB
    mkl_random-1.0.1           |   py37h77b88f5_1         267 KB
    intel-openmp-2019.0        |              117         1.7 MB
    qt-5.9.6                   |   vc14h62aca36_0        92.5 MB
    matplotlib-2.2.3           |   py37hd159220_0         6.5 MB
    tornado-5.1                |   py37hfa6e2cd_0         668 KB
    pyqt-5.9.2                 |   py37ha878b3d_0         4.6 MB
    pytz-2018.5                |           py37_0         232 KB
    scipy-1.1.0                |   py37h4f6bf74_1        13.5 MB
    jpeg-9b                    |       hb83a4c4_2         313 KB
    python-dateutil-2.7.3      |           py37_0         260 KB
    numpy-base-1.15.1          |   py37h8128ebf_0         3.9 MB
    numpy-1.15.1               |   py37ha559c80_0          37 KB
    mkl_fft-1.0.4              |   py37h1e22a9b_1         120 KB
    kiwisolver-1.0.1           |   py37h6538335_0          61 KB
    pyparsing-2.2.0            |           py37_1          96 KB
    cycler-0.10.0              |           py37_0          13 KB
    freetype-2.9.1             |       ha9979f8_1         470 KB
    icu-58.2                   |       ha66f8fd_1        21.9 MB
    sqlite-3.24.0              |       h7602738_0         899 KB
    sip-4.19.12                |   py37h6538335_0         283 KB
    ------------------------------------------------------------
                                           Total:       149.5 MB

The following NEW packages will be INSTALLED:

    blas:            1.0-mkl
    cycler:          0.10.0-py37_0
    freetype:        2.9.1-ha9979f8_1
    icc_rt:          2017.0.4-h97af966_0
    icu:             58.2-ha66f8fd_1
    intel-openmp:    2019.0-117
    jpeg:            9b-hb83a4c4_2
    kiwisolver:      1.0.1-py37h6538335_0
    libpng:          1.6.34-h79bbb47_0
    matplotlib:      2.2.3-py37hd159220_0
    mkl:             2019.0-117
    mkl_fft:         1.0.4-py37h1e22a9b_1
    mkl_random:      1.0.1-py37h77b88f5_1
    numpy:           1.15.1-py37ha559c80_0
    numpy-base:      1.15.1-py37h8128ebf_0
    pyparsing:       2.2.0-py37_1
    pyqt:            5.9.2-py37ha878b3d_0
    python-dateutil: 2.7.3-py37_0
    pytz:            2018.5-py37_0
    qt:              5.9.6-vc14h62aca36_0
    scipy:           1.1.0-py37h4f6bf74_1
    sip:             4.19.12-py37h6538335_0
    sqlite:          3.24.0-h7602738_0
    tornado:         5.1-py37hfa6e2cd_0
    zlib:            1.2.11-h8395fce_2

Proceed ([y]/n)?

Now that you’ve covered how to search and install packages, let’s see how to update and remove them using Conda.

Updating and Removing Packages

Sometimes, when new packages are released, you need to update them. To do so, you may run conda update <package name>. In case you wish to update all the packages within one environment, you should activate the environment and run conda update --all.

To remove a package, you can run conda remove <package name>. For example, this is how you remove numpy from the root base environment:

(base) C:\Users\IEUser>conda remove numpy
Solving environment: done

## Package Plan ##

  environment location: C:\Users\IEUser\Miniconda3

  removed specs:
    - numpy


The following packages will be REMOVED:

    matplotlib: 2.2.3-py37hd159220_0
    mkl_fft:    1.0.4-py37h1e22a9b_1
    mkl_random: 1.0.1-py37h77b88f5_1
    numpy:      1.15.1-py37ha559c80_0
    scipy:      1.1.0-py37h4f6bf74_1

Proceed ([y]/n)?

It’s worth noting that when you remove a package, all packages that depend on it are also removed.

Using Channels

Sometimes, you won’t find the packages you want to install on the default channels configured by the installer. For example, this is how you install pytorch, another machine learning package:

(base) C:\Users\IEUser>conda search pytorch
Loading channels: done

PackagesNotFoundError: The following packages are not available from current channels:

  - pytorch

Current channels:

  - https://repo.anaconda.com/pkgs/main/win-64
  - https://repo.anaconda.com/pkgs/main/noarch
  - https://repo.anaconda.com/pkgs/free/win-64
  - https://repo.anaconda.com/pkgs/free/noarch
  - https://repo.anaconda.com/pkgs/r/win-64
  - https://repo.anaconda.com/pkgs/r/noarch
  - https://repo.anaconda.com/pkgs/pro/win-64
  - https://repo.anaconda.com/pkgs/pro/noarch
  - https://repo.anaconda.com/pkgs/msys2/win-64
  - https://repo.anaconda.com/pkgs/msys2/noarch

To search for alternate channels that may provide the conda package you're
looking for, navigate to

    https://anaconda.org

and use the search bar at the top of the page.

In this case, you may search for the package here. If you search for pytorch, you’ll get the following results:

The channel pytorch has a package named pytorch with version 0.4.1. To install a package from a specific channel you can use the -c <channel> parameter with conda install:

(base) C:\Users\IEUser>conda install -c pytorch pytorch
Solving environment: done

## Package Plan ##

  environment location: C:\Users\IEUser\Miniconda3

  added / updated specs:
    - pytorch


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    pytorch-0.4.1              |py37_cuda90_cudnn7he774522_1       590.4 MB  pytorch

The following NEW packages will be INSTALLED:

    pytorch: 0.4.1-py37_cuda90_cudnn7he774522_1 pytorch

Proceed ([y]/n)?

Alternatively, you can add the channel, so that Conda uses it to search for packages to install. To list the current channels used, you can run conda config --get channels:

(base) C:\Users\IEUser>conda config --get channels
--add channels 'defaults'   # lowest priority

(base) C:\Users\IEUser>

The Miniconda installer includes only the defaults channels. When more channels are included, it is necessary to set the priority of them to determine from which channel a package will be installed in case it is available from more than one channel.

To add a channel with the lowest priority to the list, you should run conda config --append channels <channel name>. To add a channel with the highest priority to the list, you should run conda config --prepend channels <channel name>. It is recommended to add new channels with low priority, to keep using the default channels prior to the others. So, alternatively, you can install pytorch, adding the pytorch channel and running conda install pytorch:

(base) C:\Users\IEUser>conda config --append channels pytorch

(base) C:\Users\IEUser>conda config --get channels
--add channels 'pytorch'   # lowest priority
--add channels 'defaults'   # highest priority

(base) C:\Users\IEUser>conda install pytorch
Solving environment: done

## Package Plan ##

  environment location: C:\Users\IEUser\Miniconda3

  added / updated specs:
    - pytorch


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    pytorch-0.4.1              |py37_cuda90_cudnn7he774522_1       590.4 MB  pytorch

The following NEW packages will be INSTALLED:

    pytorch: 0.4.1-py37_cuda90_cudnn7he774522_1 pytorch

Proceed ([y]/n)?

Not all packages are available on Conda channels. However, this is not a problem, since you also can use pip to install packages inside Conda environments. Let’s see how to do this.

Using `pip` Inside Conda Environments

Sometimes, you may need pure Python packages and, generally, these packages are not available on Conda’s channels. For example, if you search for unipath, a package to deal with file paths in Python, Conda won’t be able to find it.

You could search for the package here and use another channel to install it. However, since unipath is a pure Python package, you could use pip to install it, as you would do on a regular Python setup. The only difference is that you should use pip installed by the Conda package pip. To illustrate that, let’s create a new environment called newproject. As mentioned before, you can do this running conda create:

conda create --name newproject

Next, to have pip installed, you should activate the environment and install the Conda package pip:

(base) C:\Users\IEUser>conda activate newproject

(newproject) C:\Users\IEUser>conda install pip
Solving environment: done

## Package Plan ##

  environment location: C:\Users\IEUser\Miniconda3\envs\newproject

  added / updated specs:
    - pip


The following NEW packages will be INSTALLED:

    certifi:        2018.8.24-py37_1
    pip:            10.0.1-py37_0
    python:         3.7.0-hea74fb7_0
    setuptools:     40.2.0-py37_0
    vc:             14-h0510ff6_3
    vs2015_runtime: 14.0.25123-3
    wheel:          0.31.1-py37_0
    wincertstore:   0.2-py37_0

Proceed ([y]/n)?

Finally, use pip to install the package unipath:

(newproject) C:\Users\IEUser>pip install unipath
Collecting unipath
Installing collected packages: unipath
Successfully installed unipath-1.1
You are using pip version 10.0.1, however version 18.0 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.

(newproject) C:\Users\IEUser>

After installation, you can list the installed packages with conda list and check that Unipath was installed using pip:

(newproject) C:\Users\IEUser>conda list
# packages in environment at C:\Users\IEUser\Miniconda3\envs\newproject:
#
# Name                    Version                   Build  Channel
certifi                   2018.8.24                py37_1
pip                       10.0.1                   py37_0
python                    3.7.0                hea74fb7_0
setuptools                40.2.0                   py37_0
Unipath                   1.1                       <pip>
vc                        14                   h0510ff6_3
vs2015_runtime            14.0.25123                    3
wheel                     0.31.1                   py37_0
wincertstore              0.2                      py37_0

(newproject) C:\Users\IEUser>

It’s also possible to install packages from a version control system (VCS) using pip. For example, let’s install supervisor, version 4.0.0dev0, available in a Git repository. As Git is not installed in the newproject environment, you should install it first:

(newproject) C:\Users\IEUser> conda install git

Then, install supervisor, using pip to install it from the Git repository:

(newproject) pip install -e git://github.com/Supervisor/supervisor@abef0a2be35f4aae4a4edeceadb7a213b729ef8d#egg=supervisor

After the installation finishes, you can see that supervisor is listed in the installed packages list:

(newproject) C:\Users\IEUser>conda list
#
# Name                    Version                   Build  Channel
certifi                   2018.8.24                py37_1
git                       2.18.0               h6bb4b03_0
meld3                     1.0.2                     <pip>
pip                       10.0.1                   py37_0
python                    3.7.0                hea74fb7_0
setuptools                40.2.0                   py37_0
supervisor                4.0.0.dev0                <pip>

... (more)

Now that you know the basics of using environments and managing packages with Conda, let’s create a simple machine learning example to solve a classic problem using a neural network.

A Simple Machine Learning Example

In this section, you’ll set up the environment using Conda and train a neural network to function like an XOR gate.

An XOR gate implements the digital logic exclusive OR operation, which is widely used in digital systems. It takes two digital inputs, that can be equal to 0, representing a digital false value or 1, representing a digital true value and outputs 1 (true) if the inputs are different or 0 (false), if the inputs are equal. The following table (referred as a truth table in the digital systems terminology) summarizes the XOR gate operation:

Input A	Input B	Output: A XOR B
0	0	0
0	1	1
1	0	1
1	1	0

The XOR operation can be interpreted as a classification problem, given that it takes two inputs and should classify them in one of two classes represented by 0 or 1, depending on whether the inputs are equal to each other or different from one another.

It is commonly used as a first example to train a neural network because it is simple and, at the same time, demands a nonlinear classifier, such as a neural network. The neural network will use only the data from the truth table, without knowledge about where it came from, to “learn” the operation performed by the XOR gate.

To implement the neural network, let’s create a new Conda environment, named nnxor:

(base) C:\Users\IEUser>conda create nnxor

Then, let’s activate it and install the package keras:

(base) C:\Users\IEUser>conda activate nnxor

(nnxor) C:\Users\IEUser>conda install keras

keras is a high-level API that makes easy-to-implement neural networks on top of well-known machine learning libraries, such as TensorFlow.

You’ll train the following neural network to act as an XOR gate:

The network takes two inputs, A and B, and feeds them to two neurons, represented by the big circles. Then, it takes the outputs of these two neurons and feeds them to an output neuron, which should provide the classification according to the XOR truth table.

In brief, the training process consists of adjusting the values of the weights w_1 until w_6, so that the output is consistent with the XOR truth table. To do so, input examples will be fed, one at a time, the output will be calculated according to current values of the weights and, by comparing the output with the desired output, given by the truth table, the values of the weights will be adjusted in a step-by-step process.

To organize the project, you’ll create a folder named nnxor within Windows user’s folder (C:\Users\IEUser) with a file named nnxor.py to store the Python program to implement the neural network:

In the nnxor.py file, you’ll define the network, perform the training, and test it:

import numpy as np
np.random.seed(444)

from keras.models import Sequential
from keras.layers.core import Dense, Activation
from keras.optimizers import SGD

X = np.array([[0, 0], 
              [0, 1],
              [1, 0],
              [1, 1]])
y = np.array([[0], [1], [1], [0]])

model = Sequential()
model.add(Dense(2, input_dim=2))
model.add(Activation('sigmoid'))
model.add(Dense(1))
model.add(Activation('sigmoid'))

sgd = SGD(lr=0.1)
model.compile(loss='mean_squared_error', optimizer=sgd)

model.fit(X, y, batch_size=1, epochs=5000)

if __name__ == '__main__':
    print(model.predict(X))

First, you import numpy, initialize a random seed, so that you can reproduce the same results when running the program again, and import the keras objects you’ll use to build the neural network.

Then, you define an X array, containing the 4 possible A-B sets of inputs for the XOR operation and a y array, containing the outputs for each of the sets of inputs defined in X.

The next five lines define the neural network. The Sequential() model is one of the models provided by keras to define a neural network, in which the layers of the network are defined in a sequential way. Then you define the first layer of neurons, composed of two neurons, fed by two inputs, defining their activation function as a sigmoid function in the sequence. Finally, you define the output layer composed of one neuron with the same activation function.

The following two lines define the details about the training of the network. To adjust the weights of the network, you’ll use the Stochastic Gradient Descent (SGD) with the learning rate equal to 0.1, and you’ll use the mean squared error as a loss function to be minimized.

Finally, you perform the training by running the fit() method, using X and y as training examples and updating the weights after every training example is fed into the network (batch_size=1). The number of epochs represents the number of times the whole training set will be used to train the neural network.

In this case, you’re repeating the training 5000 times using a training set containing 4 input-output examples. By default, each time the training set is used, the training examples are shuffled.

On the last line, after the training process has finished, you print the predicted values for the 4 possible input examples.

By running this script, you’ll see the evolution of the training process and the performance improvement as new training examples are fed into the network:

(nnxor) C:\Users\IEUser>cd nnxor

(nnxor) C:\Users\IEUser\nnxor>python nnxor.py
Using TensorFlow backend.
Epoch 1/5000
2018-09-16 09:49:05.987096: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2
2018-09-16 09:49:05.993128: I tensorflow/core/common_runtime/process_util.cc:69] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.
4/4 [==============================] - 0s 39ms/step - loss: 0.2565
Epoch 2/5000
4/4 [==============================] - 0s 0us/step - loss: 0.2566
Epoch 3/5000
4/4 [==============================] - 0s 0us/step - loss: 0.2566
Epoch 4/5000
4/4 [==============================] - 0s 0us/step - loss: 0.2566
Epoch 5/5000
4/4 [==============================] - 0s 0us/step - loss: 0.2566
Epoch 6/5000
4/4 [==============================] - 0s 0us/step - loss: 0.2566

After the training finishes, you can check the predictions the network gives for the possible input values:

Epoch 4997/5000
4/4 [==============================] - 0s 0us/step - loss: 0.0034
Epoch 4998/5000
4/4 [==============================] - 0s 0us/step - loss: 0.0034
Epoch 4999/5000
4/4 [==============================] - 0s 0us/step - loss: 0.0034
Epoch 5000/5000
4/4 [==============================] - 0s 0us/step - loss: 0.0034
[[0.0587215 ]
 [0.9468337 ]
 [0.9323144 ]
 [0.05158457]]

As you defined X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]]), the expected output values are 0, 1, 1, and 0, which is consistent with the predicted outputs of the network, given you should round them to obtain binary values.

Where To Go From Here

Data science and machine learning applications are emerging in the most diverse areas, attracting more people. However, setting up an environment for numerical computation can be a complicated task, and it’s common to find users having trouble in data science workshops, especially when using Windows.

In this article, you’ve covered the basics of setting up a Python numerical computation environment on a Windows machine using the Anaconda Python distribution.

Free Bonus: Click here to get access to a Conda cheat sheet with handy usage examples for managing your Python environment and packages.

Now that you have a working environment, it’s time to start working with some applications. Python is one of the most used languages for data science and machine learning, and Anaconda is one of the most popular distributions, used in various companies and research laboratories. It provides several packages to install libraries that Python relies on for data acquisition, wrangling, processing, and visualization.

Fortunately there are a lot of tutorials about these libraries available at Real Python, including the following:

Also, if you’d like a deeper understanding of Anaconda and Conda, check out the following links:

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python Community Interview With Michael Kennedy

2018-10-29T14:00:00+00:00

This week, our Python community interview is with none other than Michael Kennedy of Talk Python to Me fame.

You may know his authoritative voice, but do you know his Python story? Read on to find out about his journey with Python, what he thinks about when stuck in traffic, and his love for two wheels.

Ricky: Welcome to Real Python! If I recall correctly, you started out as a .NET developer, and you were even a Microsoft Certified Trainer. So I’m curious as to how you came to Python and what made you stick around?

Michael: Thanks for having me. Oh, this brings back memories. Yes, I was doing full time .NET development with C# for probably 10 years. It’s a language I still respect today.

I found my way to Python after wanting to branch out into areas outside of the Microsoft space. I guess this was probably 2012 or around then. I hadn’t been doing much outside of C++ and C# before then for some time other than JavaScript. (No one escapes JavaScript!) I looked at the popular languages, and this was just around the time Python was becoming popular and increasingly so.

I spent a few weeks learning Python and was pretty much hooked but didn’t know it.

I studied the language and ecosystem and found it to be really nice—much nicer than I expected. But I suffered the problem that everyone who knows some language really well does when they try something different. Everything I knew by heart was a challenge again. How do I create a web app? How do I host it? How do I query a database? And on and on.

I was willing to learn and did. But it was just that uneasiness that anyone would have, giving up the familiar path. However, I knew I was going to be hooked when I went back to write some C# code and found it way less tasteful than I had just a few weeks prior.

This is not to bash that language. But like all C-based languages, it had a lot of symbol noise, let’s say. Why do we need semicolons again? Why the massive love for parentheses and curly braces even when it’s (now) clear they are unneeded, etc. Here’s an example:

class Thing {

    public int TotalSales {
        get 
        {
            int total = 0;
            foreach (var b in items) 
            {
                total += b.value;
            }
            return total;
        }
    }
}

class Thing:

    @property
    def total_sales(self):
        total = 0
        for b in items:
            total += b.value
        return total

Makes one wonder why you’ve been typing all those symbols, all those years.

Since then, as I’ve learned more and more of the popular packages and standard library modules, I’ve just enjoyed it more every day. Now I run my entire business on a Python stack, and it has yet to let me down.

Ricky: You are, of course, the host of the most popular Python podcast—Talk Python to Me—which now has over 180 episodes. You’re also a co-host on the Python Bytes podcast with Brian Okken. That’s a lot of content! How do you continue to be so consistent each week, and keep the shows relevant and newsworthy? It must be a lot of work?

Michael: That is definitely a lot of content. But it’s a very rewarding project that is into its fourth year for Talk Python To Me, and third year for Python Bytes.

How am I still consistent? That’s a good question. I started the podcast because others were inconsistent. There have been Python-based podcasts before mine. But they all stopped producing episodes long before I got in the game. In fact, that’s why I felt I could get started, because there was such a lack of content.

I am consistent for a few reasons. First, when I started the podcast, I promised myself I’d do it every week for six months and then decide whether the community and I enjoyed it. After that much consistent content creation, you are pretty deep within the habit of doing so.

Second, by then I had several companies sponsoring the podcast. I thought maybe, just maybe, I could find a way to use the podcast to become independent of my day job. I didn’t hate my day job, but it doesn’t compare to doing what you think is truly valuable for the community and world. Once you accept money to produce a thing over a long period of time, consistency is just part of the agreement.

Finally, the listeners were so supportive of my work. It genuinely felt great producing the content for everyone. I looked forward to each episode I created. After all, I was learning so much from each one and continue to do so to this day.

As for keeping the show relevant and newsworthy, that is easy. For Python Bytes, that’s literally the topic (weekly news items), and we get tons of help from listeners all suggesting great new items each week.

For Talk Python To Me, this is harder. Each episode digs deeply into a topic. For the first 20 episodes or so, that was easy enough for me. I’d used SQLAlchemy for example, so asking Mike Bayer about it was just thinking back on my experience. But it quickly grew into spaces I had little experience with. I now spend quite a bit of time researching topics to cover each week. Any given episode has between 4–8 hours of research before we even press record.

That leads into the final part of your question: Yes, it is a lot of work. I’ve had folks ask me how much time I spend on the show each week. They’ve even said, “You have such a sweet deal. What do you spend on the podcast per episode? A few hours?” Well, that would be something! I probably spend about 2 days per episode between the research, outreach to guests, email correspondence, website development, sponsorship relationships, and much more.

That is a lot of time, but it is also literally the foundation of my business. The podcasts and the courses only work if they are both well known and high quality. It’s very fortunate that I’ve been able to transition my part-time podcast into a full-time job (podcast and courses). It lets me really stay focused and stay consistent.

Ricky: If our readers don’t know you from the podcast, they surely will know you for your excellent courses on Talk Python Training. One of the first courses I took when I began learning Python was your Python Jumpstart by Building 10 Apps course—which is excellent by the way. And you’ve just released a new course named Async Techniques and Examples. Could you tell us a little more about it and why you decided to focus on Async, specifically?

Michael: Thanks! The courses have been a true passion project for me. I’ve wanted to create the best online library for Python courses out there for a long time.

When I first started the podcast, I also wanted to start the courses. I saw them as going hand in hand, with each supporting the other. However, at the time I worked for a company that did in-person and online training courses for developers.

They afforded me a lot of freedom and flexibility. But what would not fly is my creating effectively a competing company in my spare time. So I started with the podcast, and then once I could go full time independently, my first action was to launch the training company and Python Jumpstart by Building 10 Apps on Kickstarter. That was a really fun experience and a huge success.

The new async course is super fun and something I felt really needed to exist for the community. It needs to exist for a few reasons. The Python async/concurrent programming story is a little hard to understand and make sense of, for many people. We have the GIL, which actually is covered very nicely on Real Python. This means normal threading was only effective for IO bound work. CPU bound work requires another API, multiprocessing with its own oddities and techniques.

Now, as of Python 3.5, we have the amazing async and await keywords. They are powerful and clean but add more choices and more fog to the situation. This doesn’t take into account the async features of Cython and its nogil keyword.

So the first reason is there was a lot of confusion around async and Python. You hear of people leaving Python for Go explicitly because of Go’s “better” concurrency. Usually, the type of concurrency people are looking for is IO bound, which works extremely well in Python anyway.

The next reason is that async and concurrent programming is oddly taught in the wrong order. What I mean by this is that usually lots of confusing, low-level detail is presented up front. Then finally it’s put together into examples that are useful and compelling. But the learner has to make it that far for it to pay off. This is also often paired with dire warnings of thread safety and how hard race conditions are. All of this is true and accurate. But why start there?

I wanted a course that shows how productive, fun, and actually easy async is for many cases. Once the student sees the value, then you can dive into things like thread safety and so on.

Finally, there really just are not many async courses for Python out there. I only know of one other one, and it’s behind a subscription wall.

Ricky: It’s no secret that you’re a big fan of MongoDB. What do you find most appealing about it? And if someone has never used it before, why might they consider using it with their next Python project?

Michael: I am a big fan of MongoDB. Long ago, I was complaining to a friend about how painful deploying relational database apps was. About how it’s a pain to apply the migration scripts without downtime and things like that. He said, “Well, why don’t you just use MongoDB, and you won’t have that problem?”.

I took his advice, and he was right! Since then, I’ve launched 4 or 5 major projects on MongoDB. Both of my podcast sites (talkpython.fm and pythonbytes.fm) and the course site run on MongoDB.

I know some folks have had bad experiences with MongoDB. There were a few “best practices” that were not the default in MongoDB in the early days, and there are lots of stories about these. Most of them have been fixed, and if you know to avoid them, you’re in good shape. The one major gotcha still out there is that MongoDB runs without authentication unless you explicitly create an account.

That said, MongoDB has been totally bulletproof for me and my projects. It’s been powering these sites mentioned above for years without any downtime. I have literally never run a migration or upgrade script to deploy a new schema or data model.

Mongo’s flexible schema model and MongoEngine’s class-based ODM (think ORM for MongoDB) are just right for my projects. The websites run with super low latency. It’s pretty common to get 10–20 ms response time for non-trivial pages (from request to response out of the server).

I personally can totally recommend MongoDB for your projects. I also created a free course if people are interested at freemongodbcourse.com.

Ricky: Now the Python is out of the way, it’s time to talk about the fun stuff… Math! You have a master’s degree in Mathematics, and you did start your Ph.D. Any plans to finish it in the future, or has that ship sailed? I would imagine you still have a passion for it. Do you get to scratch that itch on a daily basis?

Michael: It’s sailed, over the horizon, and halfway to Antarctica! I did study math and still very much appreciate the beauty of everything about it. I was just thinking about the different types of infinity, the different sizes of infinity, on just the simple number line between [0, 1] while stuck in traffic last week.

But that’s not working in math, day to day. I believe software development is my true calling. I love doing it every day. What I learned in math was excellent preparation for development. The rules of mathematics and the “rules” (language, APIs, algorithms, big-O, etc.) of software are surprisingly similar. The types of thinking and problem solving are also quite comparable.

The career opportunity in software and entrepreneurship vs. mathematics is not comparable. It’s just better to build things people can use (software) rather than theories only 5–20 people in the world will understand and care about (math these days).

So I love it and still read books about it, but I’m not doing anything practical with math these days.

Ricky: Do you have any other hobbies or interests, aside from Python?

Michael: I’ve had many fun hobbies over the years. I do think it’s important to have a balance between computer time and other things in your life.

I’m pretty fortunate in that I basically have 3 really engaging aspects to my job that I would almost consider hobbies. I run the podcast and am really into improving that craft and connecting with the listeners. I am doing software development still almost daily. And running my business and the whole entrepreneurial side of things is amazing and fun.

In terms of actual hobbies, I love racing and anything with two wheels! I grew up racing BMX bikes in grades 1–5, then motocross through middle school and high school, and finally mountain bikes in college. My brothers and I built a motocross track in our backyard, and it was pretty common to come home from school, drop our backpacks, and spend an hour or two challenging each other to clear this series of jumps or just having a fun time.

These days, I only watch motocross and am also a huge fan of IndyCar. I still ride but keep it to mellow adventures with my wife on our street motorcycles around the mountains here in Portland, OR. It’s great to be able to share that experience of riding with her and my daughters, who jump on the back of one of our bikes.

Ricky: What do we have to look forward to in the future from Talk Python? Any secret projects you’d like to tell us about or anything you’d like to share and/or plug?

Michael: I have some projects involving exciting courses coming out. I’d actually like to be more forthcoming with what I’m working on there, but there is a surprising amount of “copying” my popular courses let’s say.

We have at least 4 courses in active development right now and a massive list of things we’d like to build. So in terms of courses, just expect us to keep working on new ones that we see the need for in the community. You can also expect some more world-class authors there creating content. I’m really honored to be able to work on the courses for everyone and make my dream of this resource and business a reality.

In terms of the podcast, no slowing down there. We have Talk Python and Python Bytes, and both are going strong. I just hope to bring better and deeper stories to the community on Talk Python and stay on top of the most exciting programming language out there with weekly updates on Python Bytes with my co-host Brian Okken.

Thank you all for having me here on Real Python. I’m a big fan of the resource you all have created. If readers are interested in my projects, please subscribe to the podcasts at talkpython.fm and pythonbytes.fm. If they have aspects of Python they’d like to learn either personally or for their team, check out our 100+ hours of courses at training.talkpython.fm.

Thank you, Michael, for joining me for this week’s interview. It’s been great having you on the other side of the interview mic.

As always, if there is someone you would like me to interview in the future, reach out to me in the comments below, or send me a message on Twitter.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Practical Text Classification With Python and Keras

2018-10-24T14:00:00+00:00

Imagine you could know the mood of the people on the Internet. Maybe you are not interested in its entirety, but only if people are today happy on your favorite social media platform. After this tutorial, you’ll be equipped to do this. While doing this, you will get a grasp of current advancements of (deep) neural networks and how they can be applied to text.

Reading the mood from text with machine learning is called sentiment analysis, and it is one of the prominent use cases in text classification. This falls into the very active research field of natural language processing (NLP). Other common use cases of text classification include detection of spam, auto tagging of customer queries, and categorization of text into defined topics. So how can you do this?

Free Bonus: 5 Thoughts On Python Mastery, a free course for Python developers that shows you the roadmap and the mindset you'll need to take your Python skills to the next level.

Choosing a Data Set

Before we start, let’s take a look at what data we have. Go ahead and download the data set from the Sentiment Labelled Sentences Data Set from the UCI Machine Learning Repository.

By the way, this repository is a wonderful source for machine learning data sets when you want to try out some algorithms. This data set includes labeled reviews from IMDb, Amazon, and Yelp. Each review is marked with a score of 0 for a negative sentiment or 1 for a positive sentiment.

Extract the folder into a data folder and go ahead and load the data with Pandas:

import pandas as pd

filepath_dict = {'yelp':   'data/sentiment_analysis/yelp_labelled.txt',
                 'amazon': 'data/sentiment_analysis/amazon_cells_labelled.txt',
                 'imdb':   'data/sentiment_analysis/imdb_labelled.txt'}

df_list = []
for source, filepath in filepath_dict.items():
    df = pd.read_csv(filepath, names=['sentence', 'label'], sep='\t')
    df['source'] = source  # Add another column filled with the source name
    df_list.append(df)

df = pd.concat(df_list)
print(df.iloc[0])

The result will be as follows:

sentence    Wow... Loved this place.
label                              1
source                          yelp
Name: 0, dtype: object

This looks about right. With this data set, you are able to train a model to predict the sentiment of a sentence. Take a quick moment to think about how you would go about predicting the data.

One way you could do this is to count the frequency of each word in each sentence and tie this count back to the entire set of words in the data set. You would start by taking the data and creating a vocabulary from all the words in all sentences. The collection of texts is also called a corpus in NLP.

The vocabulary in this case is a list of words that occurred in our text where each word has its own index. This enables you to create a vector for a sentence. You would then take the sentence you want to vectorize, and you count each occurrence in the vocabulary. The resulting vector will be with the length of the vocabulary and a count for each word in the vocabulary.

The resulting vector is also called a feature vector. In a feature vector, each dimension can be a numeric or categorical feature, like for example the height of a building, the price of a stock, or, in our case, the count of a word in a vocabulary. These feature vectors are a crucial piece in data science and machine learning, as the model you want to train depends on them.

Let’s quickly illustrate this. Imagine you have the following two sentences:

>>>

>>> sentences = ['John likes ice cream', 'John hates chocolate.']

Next, you can use the CountVectorizer provided by the scikit-learn library to vectorize sentences. It takes the words of each sentence and creates a vocabulary of all the unique words in the sentences. This vocabulary can then be used to create a feature vector of the count of the words:

>>>

>>> from sklearn.feature_extraction.text import CountVectorizer

>>> vectorizer = CountVectorizer(min_df=0, lowercase=False)
>>> vectorizer.fit(sentences)
>>> vectorizer.vocabulary_
{'John': 0, 'chocolate': 1, 'cream': 2, 'hates': 3, 'ice': 4, 'likes': 5}

This vocabulary serves also as an index of each word. Now, you can take each sentence and get the word occurrences of the words based on the previous vocabulary. The vocabulary consists of all five words in our sentences, each representing one word in the vocabulary. When you take the previous two sentences and transform them with the CountVectorizer you will get a vector representing the count of each word of the sentence:

>>>

>>> vectorizer.transform(sentences).toarray()
array([[1, 0, 1, 0, 1, 1],
    [1, 1, 0, 1, 0, 0]])

Now, you can see the resulting feature vectors for each sentence based on the previous vocabulary. For example, if you take a look at the first item, you can see that both vectors have a 1 there. This means that both sentences have one occurrence of John, which is in the first place in the vocabulary.

This is considered a Bag-of-words (BOW) model, which is a common way in NLP to create vectors out of text. Each document is represented as a vector. You can use these vectors now as feature vectors for a machine learning model. This leads us to our next part, defining a baseline model.

Defining a Baseline Model

When you work with machine learning, one important step is to define a baseline model. This usually involves a simple model, which is then used as a comparison with the more advanced models that you want to test. In this case, you’ll use the baseline model to compare it to the more advanced methods involving (deep) neural networks, the meat and potatoes of this tutorial.

First, you are going to split the data into a training and testing set which will allow you to evaluate the accuracy and see if your model generalizes well. This means whether the model is able to perform well on data it has not seen before. This is a way to see if the model is overfitting.

Overfitting is when a model is trained too well on the training data. You want to avoid overfitting, as this would mean that the model mostly just memorized the training data. This would account for a large accuracy with the training data but a low accuracy in the testing data.

We start by taking the Yelp data set which we extract from our concatenated data set. From there, we take the sentences and labels. The .values returns a NumPy array instead of a Pandas Series object which is in this context easier to work with:

>>>

>>> from sklearn.model_selection import train_test_split

>>> df_yelp = df[df['source'] == 'yelp']

>>> sentences = df_yelp['sentence'].values
>>> y = df_yelp['label'].values

>>> sentences_train, sentences_test, y_train, y_test = train_test_split(
...    sentences, y, test_size=0.25, random_state=1000)

Here we will use again on the previous BOW model to vectorize the sentences. You can use again the CountVectorizer for this task. Since you might not have the testing data available during training, you can create the vocabulary using only the training data. Using this vocabulary, you can create the feature vectors for each sentence of the training and testing set:

>>>

>>> from sklearn.feature_extraction.text import CountVectorizer

>>> vectorizer = CountVectorizer()
>>> vectorizer.fit(sentences_train)

>>> X_train = vectorizer.transform(sentences_train)
>>> X_test  = vectorizer.transform(sentences_test)
>>> X_train
<750x1714 sparse matrix of type '<class 'numpy.int64'>'
    with 7368 stored elements in Compressed Sparse Row format>

You can see that the resulting feature vectors have 750 samples which are the number of training samples we have after the train-test split. Each sample has 1714 dimensions which is the size of the vocabulary. Also, you can see that we get a sparse matrix. This is a data type that is optimized for matrices with only a few non-zero elements, which only keeps track of the non-zero elements reducing the memory load.

CountVectorizer performs tokenization which separates the sentences into a set of tokens as you saw previously in the vocabulary. It additionally removes punctuation and special characters and can apply other preprocessing to each word. If you want, you can use a custom tokenizer from the NLTK library with the CountVectorizer or use any number of the customizations which you can explore to improve the performance of your model.

Note: There are a lot of additional parameters to CountVectorizer() that we forgo using here, such as adding ngrams, beacuse the goal at first is to build a simple baseline model. The token pattern itself defaults to token_pattern=’(?u)\b\w\w+\b’, which is a regex pattern that says, “a word is 2 or more Unicode word characters surrounded by word boundaries.”.

The classification model we are going to use is the logistic regression which is a simple yet powerful linear model that is mathematically speaking in fact a form of regression between 0 and 1 based on the input feature vector. By specifying a cutoff value (by default 0.5), the regression model is used for classification. You can use again scikit-learn library which provides the LogisticRegression classifier:

>>>

>>> from sklearn.linear_model import LogisticRegression

>>> classifier = LogisticRegression()
>>> classifier.fit(X_train, y_train)
>>> score = classifier.score(X_test, y_test)

>>> print("Accuracy:", score)
Accuracy: 0.796

You can see that the logistic regression reached an impressive 79.6%, but let’s have a look how this model performs on the other data sets that we have. In this script, we perform and evaluate the whole process for each data set that we have:

for source in df['source'].unique():
    df_source = df[df['source'] == source]
    sentences = df_source['sentence'].values
    y = df_source['label'].values

    sentences_train, sentences_test, y_train, y_test = train_test_split(
        sentences, y, test_size=0.25, random_state=1000)

    vectorizer = CountVectorizer()
    vectorizer.fit(sentences_train)
    X_train = vectorizer.transform(sentences_train)
    X_test  = vectorizer.transform(sentences_test)

    classifier = LogisticRegression()
    classifier.fit(X_train, y_train)
    score = classifier.score(X_test, y_test)
    print('Accuracy for {} data: {:.4f}'.format(source, score))

Here’s the result:

Accuracy for yelp data: 0.7960
Accuracy for amazon data: 0.7960
Accuracy for imdb data: 0.7487

Great! You can see that this fairly simple model achieves a fairly good accuracy. It would be interesting to see whether we are able to outperform this model. In the next part, we will get familiar with (deep) neural networks and how to apply them to text classification.

A Primer on (Deep) Neural Networks

You might have experienced some of the excitement and fear related to artificial intelligence and deep learning. You might have stumbled across some confusing article or concerned TED talk about the approaching singularity or maybe you saw the backflipping robots and you wonder whether a life in the woods seems reasonable after all.

On a lighter note, AI researchers all agreed that they did not agree with each other when AI will exceed Human-level performance. According to this paper we should still have some time left.

So you might already be curious how neural networks work. If you already are familiar with neural networks, feel free to skip to the parts involving Keras. Also, there is the wonderful Deep Learning book by Ian Goodfellow which I highly recommend if you want to dig deeper into the math. You can read the whole book online for free. In this section you will get an overview of neural networks and their inner workings, and you will later see how to use neural networks with the outstanding Keras library.

In this article, you don’t have to worry about the singularity, but (deep) neural networks play a crucial role in the latest developments in AI. It all started with a famous paper in 2012 by Geoffrey Hinton and his team, which outperformed all previous models in the famous ImageNet Challenge.

The challenge could be considered the World Cup in computer vision which involves classifying a large set of images based on given labels. Geoffrey Hinton and his team managed to beat the previous models by using a convolutional neural network (CNN), which we will cover in this tutorial as well.

Since then, neural networks have moved into several fields involving classification, regression and even generative models. The most prevalent fields include computer vision, voice recognition and natural language processing (NLP).

Neural networks, or sometimes called artificial neural network (ANN) or feedforward neural network, are computational networks which were vaguely inspired by the neural networks in the human brain. They consist of neurons (also called nodes) which are connected like in the graph below.

You start by having a layer of input neurons where you feed in your feature vectors and the values are then feeded forward to a hidden layer. At each connection, you are feeding the value forward, while the value is multiplied by a weight and a bias is added to the value. This happens at every connection and at the end you reach an output layer with one or more output nodes.

If you want to have a binary classification you can use one node, but if you have multiple categories you should use multiple nodes for each category:

Neural network model

You can have as many hidden layers as you wish. In fact, a neural network with more than one hidden layer is considered a deep neural network. Don’t worry: I won’t get here into the mathematical depths concerning neural networks. But if you want to get an intuitive visual understanding of the math involved, you can check out the YouTube Playlist by Grant Sanderson. The formula from one layer to the next is this short equation:

Neural network formula

Let’s slowly unpack what is happening here. You see, we are dealing here with only two layers. The layer with nodes a serves as input for the layer with nodes o. In order to calculate the values for each output node, we have to multiply each input node by a weight w and add a bias b.

All of those have to be then summed and passed to a function f. This function is considered the activation function and there are various different functions that can be used depending on the layer or the problem. It is generally common to use a rectified linear unit (ReLU) for hidden layers, a sigmoid function for the output layer in a binary classification problem, or a softmax function for the output layer of multi-class classification problems.

You might already wonder how the weights are calculated, and this is obviously the most important part of neural networks, but also the most difficult part. The algorithm starts by initializing the weights with random values and they are then trained with a method called backpropagation.

This is done by using optimization methods (also called optimizer) like the gradient descent in order to reduce the error between the computed and the desired output (also called target output). The error is determined by a loss function whose loss we want to minimize with the optimizer. The whole process is too extensive to cover here, but I’ll refer again to the Grant Sanderson playlist and the Deep Learning book by Ian Goodfellow I mentioned before.

What you have to know is that there are various optimization methods that you can use, but the most common optimizer currently used is called Adam which has a good performance in various problems.

You can also use different loss functions, but in this tutorial you will only need the cross entropy loss function or more specifically binary cross entropy which is used for binary classification problems. Be sure to experiment with the various available methods and tools. Some researchers even claim in a recent article that the choice for the best performing methods borders on alchemy. The reason being that many methods are not well explained and consist of a lot of tweaking and testing.

Introducing Keras

Keras is a deep learning and neural networks API by François Chollet which is capable of running on top of Tensorflow (Google), Theano or CNTK (Microsoft). To quote the wonderful book by François Chollet, Deep Learning with Python:

Keras is a model-level library, providing high-level building blocks for developing deep-learning models. It doesn’t handle low-level operations such as tensor manipulation and differentiation. Instead, it relies on a specialized, well-optimized tensor library to do so, serving as the backend engine of Keras (Source)

It is a great way to start experimenting with neural networks without having to implement every layer and piece on your own. For example Tensorflow is a great machine learning library, but you have to implement a lot of boilerplate code to have a model running.

Installing Keras

Before installing Keras, you’ll need either Tensorflow, Theano, or CNTK. In this tutorial we will be using Tensorflow so check out their installation guide here, but feel free to use any of the frameworks that works best for you. Keras can be installed using PyPI with the following command:

$ pip install keras

You can choose the backend you want to have by opening the Keras configuration file which you can find here:

$HOME/.keras/keras.json

If you are a Windows user, you have to replace $HOME with %USERPROFILE%. The configuration file should look as follows:

{
    "image_data_format": "channels_last",
    "epsilon": 1e-07,
    "floatx": "float32",
    "backend": "tensorflow"
}

You can change the backend field there to "theano", "tensorflow" or "cntk", given that you have installed the backend on your machine. For more details check out the Keras backends documentation.

You might notice that we use float32 data in the configuration file. The reason for this is that neural networks are frequently used in GPUs, and the computational bottleneck is memory. By using 32 bit, we are able reduce the memory load and we do not lose too much information in the process.

Your First Keras Model

Now you are finally ready to experiment with Keras. Keras supports two main types of models. You have the Sequential model API which you are going to see in use in this tutorial and the functional API which can do everything of the Sequential model but it can be also used for advanced models with complex network architectures.

The Sequential model is a linear stack of layers, where you can use the large variety of available layers in Keras. The most common layer is the Dense layer which is your regular densely connected neural network layer with all the weights and biases that you are already familiar with.

Let’s see if we can achieve some improvement to our previous logistic regression model. You can use the X_train and X_test arrays that you built in our earlier example.

Before we build our model, we need to know the input dimension of our feature vectors. This happens only in the first layer since the following layers can do automatic shape inference. In order to build the Sequential model, you can add layers one by one in order as follows:

>>>

>>> from keras.models import Sequential
>>> from keras import layers

>>> input_dim = X_train.shape[1]  # Number of features

>>> model = Sequential()
>>> model.add(layers.Dense(10, input_dim=input_dim, activation='relu'))
>>> model.add(layers.Dense(1, activation='sigmoid'))
Using TensorFlow backend.

Before you can start with the training of the model, you need to configure the learning process. This is done with the .compile() method. This method specifies the optimizer and the loss function.

Additionally, you can add a list of metrics which can be later used for evaluation, but they do not influence the training. In this case, we want to use the binary cross entropy and the Adam optimizer you saw in the primer mentioned before. Keras also includes a handy .summary() function to give an overview of the model and the number of parameters available for training:

>>>

>>> model.compile(loss='binary_crossentropy', 
...               optimizer='adam', 
...               metrics=['accuracy'])
>>> model.summary()
_________________________________________________________________
Layer (type)                 Output Shape          Param #   
=================================================================
dense_1 (Dense)              (None, 10)            17150     
_________________________________________________________________
dense_2 (Dense)              (None, 1)             11        
=================================================================
Total params: 17,161
Trainable params: 17,161
Non-trainable params: 0
_________________________________________________________________

You might notice that we have 8575 parameters for the first layer and another 6 in the next one. Where did those come from?

See, we have 1714 dimensions for each feature vector, and then we have 5 nodes. We need weights for each feature dimension and each node which accounts for 1714 * 5 = 8570 parameters, and then we have another 5 times an added bias for each node, which gets us the 8575 parameters. In the final node, we have another 5 weights and one bias, which gets us to 6 parameters.

Neat! You are almost there. Now it is time to start your training with the .fit() function.

Since the training in neural networks is an iterative process, the training won’t just stop after it is done. You have to specify the number of iterations you want the model to be training. Those completed iterations are commonly called epochs. We want to run it for 100 epochs to be able to see how the training loss and accuracy are changing after each epoch.

Another parameter you have to your selection is the batch size. The batch size is responsible for how many samples we want to use in one epoch, which means how many samples are used in one forward/backward pass. This increases the speed of the computation as it need fewer epochs to run, but it also needs more memory, and the model may degrade with larger batch sizes. Since we have a small training set, we can leave this to a low batch size:

>>>

>>> history = model.fit(X_train, y_train,
...                     epochs=100,
...                     verbose=False,
...                     validation_data=(X_test, y_test)
...                     batch_size=10)

Now you can use the .evaluate() method to measure the accuracy of the model. You can do this both for the training data and testing data. We expect that the training data has a higher accuracy then for the testing data. Tee longer you would train a neural network, the more likely it is that it starts overfitting.

Note that if you rerun the .fit() method, you’ll start off with the computed weights from the previous training. Make sure to compile the model again before you start training the model again. Now let’s evaluate the accuracy model:

>>>

>>> loss, accuracy = model.evaluate(X_train, y_train, verbose=False)
>>> print("Training Accuracy: {:.4f}".format(accuracy))
>>> loss, accuracy = model.evaluate(X_test, y_test, verbose=False)
>>> print("Testing Accuracy:  {:.4f}".format(accuracy))
Training Accuracy: 1.0000
Testing Accuracy:  0.7960

You can already see that the model was overfitting since it reached 100% accuracy for the training set. But this was expected since the number of epochs was fairly large for this model. However, the accuracy of the testing set has already surpassed our previous logistic Regression with BOW model, which is a great step further in terms of our progress.

To make your life easier, you can use this little helper function to visualize the loss and accuracy for the training and testing data based on the History callback. This callback, which is automatically applied to each Keras model, records the loss and additional metrics that can be added in the .fit() method. In this case, we are only interested in the accuracy. This helper function employs the matplotlib plotting library:

import matplotlib.pyplot as plt
plt.style.use('ggplot')

def plot_history(history):
    acc = history.history['acc']
    val_acc = history.history['val_acc']
    loss = history.history['loss']
    val_loss = history.history['val_loss']
    x = range(1, len(acc) + 1)

    plt.figure(figsize=(12, 5))
    plt.subplot(1, 2, 1)
    plt.plot(x, acc, 'b', label='Training acc')
    plt.plot(x, val_acc, 'r', label='Validation acc')
    plt.title('Training and validation accuracy')
    plt.legend()
    plt.subplot(1, 2, 2)
    plt.plot(x, loss, 'b', label='Training loss')
    plt.plot(x, val_loss, 'r', label='Validation loss')
    plt.title('Training and validation loss')
    plt.legend()

To use this function, simply call plot_history() with the collected accuracy and loss inside the history dictionary:

>>>

>>> plot_history(history)

Accuracy and loss for baseline model

You can see that we have trained our model for too long since the training set reached 100% accuracy. A good way to see when the model starts overfitting is when the loss of the validation data starts rising again. This tends to be a good point to stop the model. You can see this around 20-40 epochs in this training.

Note: When training neural networks, you should use a separate testing and validation set. What you would usually do is take the model with the highest validation accuracy and then test the model with the testing set.

This makes sure that you don’t overfit the model. Using the validation set to choose the best model is a form of data leakage (or “cheating”) to get to pick the result that produced the best test score out of hundreds of them. Data leakage happens when information outside the training data set is used in the model.

In this case, our testing and validation set are the same, since we have a smaller sample size. As we have covered before, (deep) neural networks perform best when you have a very large number of samples. In the next part, you’ll see a different way to represent words as vectors. This is a very exciting and powerful way to work with words where you’ll see how to represent words as dense vectors.

What Is a Word Embedding?

Text is considered a form of sequence data similar to time series data that you would have in weather data or financial data. In the previous BOW model, you have seen how to represent a whole sequence of words as a single feature vector. Now you will see how to represent each word as vectors. There are various ways to vectorize text, such as:

Words represented by each word as a vector
Characters represented by each character as a vector
N-grams of words/characters represented as a vector (N-grams are overlapping groups of multiple succeeding words/characters in the text)

In this tutorial, you’ll see how to deal with representing words as vectors which is the common way to use text in neural networks. Two possible ways to represent a word as a vector are one-hot encoding and word embeddings.

One-Hot Encoding

The first way to represent a word as a vector is by creating a so-called one-hot encoding, which is simply done by taking a vector of the length of the vocabulary with an entry for each word in the corpus.

In this way, you have for each word, given it has a spot in the vocabulary, a vector with zeros everywhere except for the corresponding spot for the word which is set to one. As you might imagine, this can become a fairly large vector for each word and it does not give any additional information like the relationship between words.

Let’s say you have a list of cities as in the following example:

>>>

>>> cities = ['London', 'Berlin', 'Berlin', 'New York', 'London']
>>> cities
['London', 'Berlin', 'Berlin', 'New York', 'London']

You can use scikit-learn and the LabelEncoder to encode the list of cities into categorical integer values like here:

>>>

>>> from sklearn.preprocessing import LabelEncoder

>>> encoder = LabelEncoder()
>>> city_labels = encoder.fit_transform(cities)
>>> city_labels
array([1, 0, 0, 2, 1])

Using this representation, you can use the OneHotEncoder provided by scikit-learn to encode the categorical values we got before into a one-hot encoded numeric array. OneHotEncoder expects each categorical value to be in a separate row, so you’ll need to reshape the array, then you can apply the encoder:

>>>

>>> from sklearn.preprocessing import OneHotEncoder

>>> encoder = OneHotEncoder(sparse=False)
>>> city_labels = city_labels.reshape((5, 1))
>>> encoder.fit_transform(city_labels)
array([[0., 1., 0.],
       [1., 0., 0.],
       [1., 0., 0.],
       [0., 0., 1.],
       [0., 1., 0.]])

You can see that categorical integer value represents the position of the array which is 1 and the rest is 0. This is often used when you have a categorical feature which you cannot represent as a numeric value but you still want to be able to use it in machine learning. One use case for this encoding is of course words in a text but it is most prominently used for categories. Such categories can be for example city, department, or other categories.

Word Embeddings

This method represents words as dense word vectors (also called word embeddings) which are trained unlike the one-hot encoding which are hardcoded. This means that the word embeddings collect more information into fewer dimensions.

Note that the word embeddings do not understand the text as a human would, but they rather map the statistical structure of the language used in the corpus. Their aim is to map semantic meaning into a geometric space. This geometric space is then called the embedding space.

This would map semantically similar words close on the embedding space like numbers or colors. If the embedding captures the relationship between words well, things like vector arithmetic should become possible. A famous example in this field of study is the ability to map King - Man + Woman = Queen.

How can you get such a word embedding? You have two options for this. One way is to train your word embeddings during the training of your neural network. The other way is by using pretrained word embeddings which you can directly use in your model. There you have the option to either leave these word embeddings unchanged during training or you train them also.

Now you need to tokenize the data into a format that can be used by the word embeddings. Keras offers a couple of convenience methods for text preprocessing and sequence preprocessing which you can employ to prepare your text.

You can start by using the Tokenizer utility class which can vectorize a text corpus into a list of integers. Each integer maps to a value in a dictionary that encodes the entire corpus, with the keys in the dictionary being the vocabulary terms themselves. You can add the parameter num_words, which is responsible for setting the size of the vocabulary. The most common num_words words will be then kept. I have the testing and training data prepared from the previous example:

>>>

>>> from keras.preprocessing.text import Tokenizer

>>> tokenizer = Tokenizer(num_words=5000)
>>> tokenizer.fit_on_texts(sentences_train)

>>> X_train = tokenizer.texts_to_sequences(sentences_train)
>>> X_test = tokenizer.texts_to_sequences(sentences_test)

>>> vocab_size = len(tokenizer.word_index) + 1  # Adding 1 because of reserved 0 index

>>> print(sentences_train[2])
>>> print(X_train[2])
Of all the dishes, the salmon was the best, but all were great.
[11, 43, 1, 171, 1, 283, 3, 1, 47, 26, 43, 24, 22]

The indexing is ordered after the most common words in the text, which you can see by the word the having the index 1. It is important to note that the index 0 is reserved and is not assigned to any word. This zero index is used for padding, which I’ll introduce in a moment.

Unknown words (words that are not in the vocabulary) are denoted in Keras with word_count + 1 since they can also hold some information. You can see the index of each word by taking a look at the word_index dictionary of the Tokenizer object:

>>>

>>> for word in ['the', 'all', 'happy', 'sad']:
...     print('{}: {}'.format(word, tokenizer.word_index[word]))
the: 1
all: 43
happy: 320
sad: 450

Note: Pay close attention to the difference between this technique and the X_train that was produced by scikit-learn’s CountVectorizer.

With CountVectorizer, we had stacked vectors of word counts, and each vector was the same length (the size of the total corpus vocabulary). With Tokenizer, the resulting vectors equal the length of each text, and the numbers don’t denote counts, but rather correspond to the word values from the dictionary tokenizer.word_index.

One problem that we have is that each text sequence has in most cases different length of words. To counter this, you can use pad_sequence() which simply pads the sequence of words with zeros. By default, it prepends zeros but we want to append them. Typically it does not matter whether you prepend or append zeros.

Additionally you would want to add a maxlen parameter to specify how long the sequences should be. This cuts sequences that exceed that number. In the following code, you can see how to pad sequences with Keras:

>>>

>>> from keras.preprocessing.sequence import pad_sequences

>>> maxlen = 100

>>> X_train = pad_sequences(X_train, padding='post', maxlen=maxlen)
>>> X_test = pad_sequences(X_test, padding='post', maxlen=maxlen)

>>> print(X_train[0, :])
[  1  10   3 282 739  25   8 208  30  64 459 230  13   1 124   5 231   8
  58   5  67   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0
   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0
   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0
   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0
   0   0   0   0   0   0   0   0   0   0]

The first values represent the index in the vocabulary as you have learned from the previous examples. You can also see that the resulting feature vector contains mostly zeros, since you have a fairly short sentence. In the next part you will see how to work with word embeddings in Keras.

Keras Embedding Layer

Notice that, at this point, our data is still hardcoded. We have not told Keras to learn a new embedding space through successive tasks. Now you can use the Embedding Layer of Keras which takes the previously calculated integers and maps them to a dense vector of the embedding. You will need the following parameters:

input_dim: the size of the vocabulary
output_dim: the size of the dense vector
input_length: the length of the sequence

With the Embedding layer we have now a couple of options. One way would be to take the output of the embedding layer and plug it into a Dense layer. In order to do this you have to add a Flatten layer in between that prepares the sequential input for the Dense layer:

from keras.models import Sequential
from keras import layers

embedding_dim = 50

model = Sequential()
model.add(layers.Embedding(input_dim=vocab_size, 
                           output_dim=embedding_dim, 
                           input_length=maxlen))
model.add(layers.Flatten())
model.add(layers.Dense(10, activation='relu'))
model.add(layers.Dense(1, activation='sigmoid'))
model.compile(optimizer='adam',
              loss='binary_crossentropy',
              metrics=['accuracy'])
model.summary()

The result will be as follows:

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
embedding_8 (Embedding)      (None, 100, 50)           87350     
_________________________________________________________________
flatten_3 (Flatten)          (None, 5000)              0         
_________________________________________________________________
dense_13 (Dense)             (None, 10)                50010     
_________________________________________________________________
dense_14 (Dense)             (None, 1)                 11        
=================================================================
Total params: 137,371
Trainable params: 137,371
Non-trainable params: 0
_________________________________________________________________

You can now see that we have 87350 new parameters to train. This number comes from vocab_size times the embedding_dim. These weights of the embedding layer are initialized with random weights and are then adjusted through backpropagation during training. This model takes the words as they come in the order of the sentences as input vectors. You can train it with the following:

history = model.fit(X_train, y_train,
                    epochs=20,
                    verbose=False,
                    validation_data=(X_test, y_test),
                    batch_size=10)
loss, accuracy = model.evaluate(X_train, y_train, verbose=False)
print("Training Accuracy: {:.4f}".format(accuracy))
loss, accuracy = model.evaluate(X_test, y_test, verbose=False)
print("Testing Accuracy:  {:.4f}".format(accuracy))
plot_history(history)

The result will be as follows:

Training Accuracy: 0.5100
Testing Accuracy:  0.4600

Accuracy and loss for first model

This is typically a not very reliable way to work with sequential data as you can see in the performance. When working with sequential data you want to focus on methods that look at local and sequential information instead of absolute positional information.

Another way to work with embeddings is by using a MaxPooling1D/AveragePooling1D or a GlobalMaxPooling1D/GlobalAveragePooling1D layer after the embedding. You can think of the pooling layers as a way to downsample (a way to reduce the size of) the incoming feature vectors.

In the case of max pooling you take the maximum value of all features in the pool for each feature dimension. In the case of average pooling you take the average, but max pooling seems to be more commonly used as it highlights large values.

Global max/average pooling takes the maximum/average of all features whereas in the other case you have to define the pool size. Keras has again its own layer that you can add in the sequential model:

from keras.models import Sequential
from keras import layers

embedding_dim = 50

model = Sequential()
model.add(layers.Embedding(input_dim=vocab_size, 
                           output_dim=embedding_dim, 
                           input_length=maxlen))
model.add(layers.GlobalMaxPool1D())
model.add(layers.Dense(10, activation='relu'))
model.add(layers.Dense(1, activation='sigmoid'))
model.compile(optimizer='adam',
              loss='binary_crossentropy',
              metrics=['accuracy'])
model.summary()

The result will be as follows:

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
embedding_9 (Embedding)      (None, 100, 50)           87350     
_________________________________________________________________
global_max_pooling1d_5 (Glob (None, 50)                0         
_________________________________________________________________
dense_15 (Dense)             (None, 10)                510       
_________________________________________________________________
dense_16 (Dense)             (None, 1)                 11        
=================================================================
Total params: 87,871
Trainable params: 87,871
Non-trainable params: 0
_________________________________________________________________

The procedure for training does not change:

history = model.fit(X_train, y_train,
                    epochs=50,
                    verbose=False,
                    validation_data=(X_test, y_test),
                    batch_size=10)
loss, accuracy = model.evaluate(X_train, y_train, verbose=False)
print("Training Accuracy: {:.4f}".format(accuracy))
loss, accuracy = model.evaluate(X_test, y_test, verbose=False)
print("Testing Accuracy:  {:.4f}".format(accuracy))
plot_history(history)

The result will be as follows:

Training Accuracy: 1.0000
Testing Accuracy:  0.8050

Accuracy and loss for max pooling model

You can already see some improvements in our models. Next you’ll see how we can employ pretrained word embeddings and if they help us with our model.

Using Pretrained Word Embeddings

We just saw an example of jointly learning word embeddings incorporated into the larger model that we want to solve.

An alternative is to use a precomputed embedding space that utilizes a much larger corpus. It is possible to precompute word embeddings by simply training them on a large corpus of text. Among the most popular methods are Word2Vec developed by Google and GloVe (Global Vectors for Word Representation) developed by the Stanford NLP Group.

Note that those are different approaches with the same goal. Word2Vec achieves this by employing neural networks and GloVe achieves this with a co-occurrence matrix and by using matrix factorization. In both cases you are dealing with dimensionality reduction, but Word2Vec is more accurate and GloVe is faster to compute.

In this tutorial, you’ll see how to work with the GloVe word embeddings from the Stanford NLP Group as their size is more manageable than the Word2Vec word embeddings provided by Google. Go ahead and download the 6B (trained on 6 billion words) word embeddings from here (822 MB).

You can find other word embeddings also on the main GloVe page. You can find the pretrained Word2Vec embeddings by Google here. If you want to train your own word embeddings, you can do so efficiently with the gensim Python package which uses Word2Vec for calculation. More details on how to do this here.

Now that we got you covered, you can start using the word embeddings in your models. You can see in the next example how you can load the embedding matrix. Each line in the file starts with the word and is followed by the embedding vector for the particular word.

This is a large file with 400000 lines, with each line representing a word followed by its vector as a stream of floats. For example, here are the first 50 characters of the first line:

$ head -n 1 data/glove_word_embeddings/glove.6B.50d.txt | cut -c-50
    the 0.418 0.24968 -0.41242 0.1217 0.34527 -0.04445

Since you don’t need all words, you can focus on only the words that we have in our vocabulary. Since we have only a limited number of words in our vocabulary, we can skip most of the 40000 words in the pretrained word embeddings:

import numpy as np

def create_embedding_matrix(filepath, word_index, embedding_dim):
    vocab_size = len(word_index) + 1  # Adding again 1 because of reserved 0 index
    embedding_matrix = np.zeros((vocab_size, embedding_dim))

    with open(filepath) as f:
        for line in f:
            word, *vector = line.split()
            if word in word_index:
                idx = word_index[word] 
                embedding_matrix[idx] = np.array(
                    vector, dtype=np.float32)[:embedding_dim]

    return embedding_matrix

You can use this function now to retrieve the embedding matrix:

>>>

>>> embedding_dim = 50
>>> embedding_matrix = create_embedding_matrix(
...     'data/glove_word_embeddings/glove.6B.50d.txt',
...     tokenizer.word_index, embedding_dim)

Wonderful! Now you are ready to use the embedding matrix in training. Let’s go ahead and use the previous network with global max pooling and see if we can improve this model. When you use pretrained word embeddings you have the choice to either allow the embedding to be updated during training or only use the resulting embedding vectors as they are.

First, let’s have a quick look how many of the embedding vectors are nonzero:

>>>

>>> nonzero_elements = np.count_nonzero(np.count_nonzero(embedding_matrix, axis=1))
>>> nonzero_elements / vocab_size
0.9507727532913566

This means 95.1% of the vocabulary is covered by the pretrained model, which is a good coverage of our vocabulary. Let’s have a look at the performance when using the GlobalMaxPool1D layer:

model = Sequential()
model.add(layers.Embedding(vocab_size, embedding_dim, 
                           weights=[embedding_matrix], 
                           input_length=maxlen, 
                           trainable=False))
model.add(layers.GlobalMaxPool1D())
model.add(layers.Dense(10, activation='relu'))
model.add(layers.Dense(1, activation='sigmoid'))
model.compile(optimizer='adam',
              loss='binary_crossentropy',
              metrics=['accuracy'])
model.summary()

The result will be as follows:

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
embedding_10 (Embedding)     (None, 100, 50)           87350     
_________________________________________________________________
global_max_pooling1d_6 (Glob (None, 50)                0         
_________________________________________________________________
dense_17 (Dense)             (None, 10)                510       
_________________________________________________________________
dense_18 (Dense)             (None, 1)                 11        
=================================================================
Total params: 87,871
Trainable params: 521
Non-trainable params: 87,350
_________________________________________________________________

history = model.fit(X_train, y_train,
                    epochs=50,
                    verbose=False,
                    validation_data=(X_test, y_test),
                    batch_size=10)
loss, accuracy = model.evaluate(X_train, y_train, verbose=False)
print("Training Accuracy: {:.4f}".format(accuracy))
loss, accuracy = model.evaluate(X_test, y_test, verbose=False)
print("Testing Accuracy:  {:.4f}".format(accuracy))
plot_history(history)

The result will be as follows:

Training Accuracy: 0.7500
Testing Accuracy:  0.6950

Accuracy and loss for untrained word embeddings

Since the word embeddings are not additionally trained, it is expected to be lower. But let’s now see how this performs if we allow the embedding to be trained by using trainable=True:

model = Sequential()
model.add(layers.Embedding(vocab_size, embedding_dim, 
                           weights=[embedding_matrix], 
                           input_length=maxlen, 
                           trainable=True))
model.add(layers.GlobalMaxPool1D())
model.add(layers.Dense(10, activation='relu'))
model.add(layers.Dense(1, activation='sigmoid'))
model.compile(optimizer='adam',
              loss='binary_crossentropy',
              metrics=['accuracy'])
model.summary()

The result will be as follows:

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
embedding_11 (Embedding)     (None, 100, 50)           87350     
_________________________________________________________________
global_max_pooling1d_7 (Glob (None, 50)                0         
_________________________________________________________________
dense_19 (Dense)             (None, 10)                510       
_________________________________________________________________
dense_20 (Dense)             (None, 1)                 11        
=================================================================
Total params: 87,871
Trainable params: 87,871
Non-trainable params: 0
_________________________________________________________________

history = model.fit(X_train, y_train,
                    epochs=50,
                    verbose=False,
                    validation_data=(X_test, y_test),
                    batch_size=10)
loss, accuracy = model.evaluate(X_train, y_train, verbose=False)
print("Training Accuracy: {:.4f}".format(accuracy))
loss, accuracy = model.evaluate(X_test, y_test, verbose=False)
print("Testing Accuracy:  {:.4f}".format(accuracy))
plot_history(history)

The result will be as follows:

Training Accuracy: 1.0000
Testing Accuracy:  0.8250

Accuracy and Loss for pretrained word embeddings

You can see that it is most effective to allow the embeddings to be trained. When dealing with large training sets it can boost the training process to be much faster than without. In our case it seemed to help but not by much. This does not have to be because of pretrained word embeddings.

Now it is time to focus on a more advanced neural network model to see if it is possible to boost the model and give it the leading edge over the previous models.

Convolutional Neural Networks (CNN)

Convolutional neural networks or also called convnets are one of the most exciting developments in machine learning in recent years.

They have revolutionized image classification and computer vision by being able to extract features from images and using them in neural networks. The properties that made them useful in image processing makes them also handy for sequence processing. You can imagine a CNN as a specialized neural network that is able to detect specific patterns.

If it is just another neural network, what differentiates it from what you have previously learned?

A CNN has hidden layers which are called convolutional layers. When you think of images, a computer has to deal with a two dimensional matrix of numbers and therefore you need some way to detect features in this matrix. These convolutional layers are able to detect edges, corners and other kinds of textures which makes them such a special tool. The convolutional layer consists of multiple filters which are slid across the image and are able to detect specific features.

This is the very core of the technique, the mathematical process of convolution. With each convolutional layer the network is able to detect more complex patterns. In the Feature Visualization by Chris Olah you can get a good intuition what these features can look like.

When you are working with sequential data, like text, you work with one dimensional convolutions, but the idea and the application stays the same. You still want to pick up on patterns in the sequence which become more complex with each added convolutional layer.

In the next figure you can see how such a convolution works. It starts by taking a patch of input features with the size of the filter kernel. With this patch you take the dot product of the multiplied weights of the filter. The one dimensional convnet is invariant to translations, which means that certain sequences can be recognized at a different position. This can be helpful for certain patterns in the text:

1D Convolution (Image source)

Now let’s have a look how you can use this network in Keras. Keras offers again various Convolutional layers which you can use for this task. The layer you’ll need is the Conv1D layer. This layer has again various parameters to choose from. The ones you are interested in for now are the number of filters, the kernel size, and the activation function. You can add this layer in between the Embedding layer and the GlobalMaxPool1D layer:

embedding_dim = 100

model = Sequential()
model.add(layers.Embedding(vocab_size, embedding_dim, input_length=maxlen))
model.add(layers.Conv1D(128, 5, activation='relu'))
model.add(layers.GlobalMaxPooling1D())
model.add(layers.Dense(10, activation='relu'))
model.add(layers.Dense(1, activation='sigmoid'))
model.compile(optimizer='adam',
              loss='binary_crossentropy',
              metrics=['accuracy'])
model.summary()

The result will be as follows:

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
embedding_13 (Embedding)     (None, 100, 100)          174700    
_________________________________________________________________
conv1d_2 (Conv1D)            (None, 96, 128)           64128     
_________________________________________________________________
global_max_pooling1d_9 (Glob (None, 128)               0         
_________________________________________________________________
dense_23 (Dense)             (None, 10)                1290      
_________________________________________________________________
dense_24 (Dense)             (None, 1)                 11        
=================================================================
Total params: 240,129
Trainable params: 240,129
Non-trainable params: 0
_________________________________________________________________

history = model.fit(X_train, y_train,
                    epochs=10,
                    verbose=False,
                    validation_data=(X_test, y_test),
                    batch_size=10)
loss, accuracy = model.evaluate(X_train, y_train, verbose=False)
print("Training Accuracy: {:.4f}".format(accuracy))
loss, accuracy = model.evaluate(X_test, y_test, verbose=False)
print("Testing Accuracy:  {:.4f}".format(accuracy))
plot_history(history)

The result will be as follows:

Training Accuracy: 1.0000
Testing Accuracy:  0.7700

Accuracy and loss for convolutional neural network

You can see that 80% accuracy seems to be tough hurdle to overcome with this data set and a CNN might not be well equipped. The reason for such a plateau might be that:

There are not enough training samples
The data you have does not generalize well
Missing focus on tweaking the hyperparameters

CNNs work best with large training sets where they are able to find generalizations where a simple model like logistic regression won’t be able.

Hyperparameters Optimization

One crucial steps of deep learning and working with neural networks is hyperparameter optimization.

As you saw in the models that we have used so far, even with simpler ones, you had a large number of parameters to tweak and choose from. Those parameters are called hyperparameters. This is the most time consuming part of machine learning and sadly there are no one-fits-all solutions ready.

When you have a look at the competitions on Kaggle, one of the largest places to compete against other fellow data scientists, you can see that many of the winning teams and models have gone through a lot of tweaking and experimenting until they reached their prime. So don’t get discouraged when it gets tough and you reach a plateau, but rather think about the ways you could optimize the model or the data.

One popular method for hyperparameter optimization is grid search. What this method does is it takes lists of parameters and it runs the model with each parameter combination that it can find. It is the most thorough way but also the most computationally heavy way to do this. Another common way, random search, which you’ll see in action here, simply takes random combinations of parameters.

In order to apply random search with Keras, you will need to use the KerasClassifier which serves as a wrapper for the scikit-learn API. With this wrapper you are able to use the various tools available with scikit-learn like cross-validation. The class that you need is RandomizedSearchCV which implements random search with cross-validation. Cross-validation is a way to validate the model and take the whole data set and separate it into multiple testing and training data sets.

There are various types of cross-validation. One type is the k-fold cross-validation which you’ll see in this example. In this type the data set is partitioned into k equal sized sets where one set is used for testing and the rest of the partitions are used for training. This enables you to run k different runs, where each partition is once used as a testing set. So, the higher k is the more accurate the model evaluation is, but the smaller each testing set is.

First step for KerasClassifier is to have a function that creates a Keras model. We will use the previous model, but we will allow various parameters to be set for the hyperparameter optimization:

def create_model(num_filters, kernel_size, vocab_size, embedding_dim, maxlen):
    model = Sequential()
    model.add(layers.Embedding(vocab_size, embedding_dim, input_length=maxlen))
    model.add(layers.Conv1D(num_filters, kernel_size, activation='relu'))
    model.add(layers.GlobalMaxPooling1D())
    model.add(layers.Dense(10, activation='relu'))
    model.add(layers.Dense(1, activation='sigmoid'))
    model.compile(optimizer='adam',
                  loss='binary_crossentropy',
                  metrics=['accuracy'])
    return model

Next, you want to define the parameter grid that you want to use in training. This consists of a dictionary with each parameters named as in the previous function. The number of spaces on the grid is 3 * 3 * 1 * 1 * 1, where each of those numbers is the number of different choices for a given parameter.

You can see how this could get computationally expensive very quickly, but luckily both grid search and random search are embarrassingly parallel, and the classes come with an n_jobs parameter that lets you test grid spaces in parallel. The parameter grid is initialized with the following dictionary:

param_grid = dict(num_filters=[32, 64, 128],
                  kernel_size=[3, 5, 7],
                  vocab_size=[5000], 
                  embedding_dim=[50],
                  maxlen=[100])

Now you are already ready to start running the random search. In this example we iterate over each data set and then you want to preprocess the data in the same way as previously. Afterwards you take the previous function and add it to the KerasClassifier wrapper class including the number of epochs.

The resulting instance and the parameter grid are then used as the estimator in the RandomSearchCV class. Additionally, you can choose the number of folds in the k-folds cross-validation, which is in this case 4. You have seen most of the code in this snippet before in our previous examples. Besides the RandomSearchCV and KerasClassifier, I have added a little block of code handling the evaluation:

from keras.wrappers.scikit_learn import KerasClassifier
from sklearn.model_selection import RandomizedSearchCV

# Main settings
epochs = 20
embedding_dim = 50
maxlen = 100
output_file = 'data/output.txt'

# Run grid search for each source (yelp, amazon, imdb)
for source, frame in df.groupby('source'):
    print('Running grid search for data set :', source)
    sentences = df['sentence'].values
    y = df['label'].values

    # Train-test split
    sentences_train, sentences_test, y_train, y_test = train_test_split(
        sentences, y, test_size=0.25, random_state=1000)

    # Tokenize words
    tokenizer = Tokenizer(num_words=5000)
    tokenizer.fit_on_texts(sentences_train)
    X_train = tokenizer.texts_to_sequences(sentences_train)
    X_test = tokenizer.texts_to_sequences(sentences_test)

    # Adding 1 because of reserved 0 index
    vocab_size = len(tokenizer.word_index) + 1

    # Pad sequences with zeros
    X_train = pad_sequences(X_train, padding='post', maxlen=maxlen)
    X_test = pad_sequences(X_test, padding='post', maxlen=maxlen)

    # Parameter grid for grid search
    param_grid = dict(num_filters=[32, 64, 128],
                      kernel_size=[3, 5, 7],
                      vocab_size=[vocab_size],
                      embedding_dim=[embedding_dim],
                      maxlen=[maxlen])
    model = KerasClassifier(build_fn=create_model,
                            epochs=epochs, batch_size=10,
                            verbose=False)
    grid = RandomizedSearchCV(estimator=model, param_distributions=param_grid,
                              cv=4, verbose=1, n_iter=5)
    grid_result = grid.fit(X_train, y_train)

    # Evaluate testing set
    test_accuracy = grid.score(X_test, y_test)

    # Save and evaluate results
    prompt = input(f'finished {source}; write to file and proceed? [y/n]')
    if prompt.lower() not in {'y', 'true', 'yes'}:
        break
    with open(output_file, 'a') as f:
        s = ('Running {} data set\nBest Accuracy : '
             '{:.4f}\n{}\nTest Accuracy : {:.4f}\n\n')
        output_string = s.format(
            source,
            grid_result.best_score_,
            grid_result.best_params_,
            test_accuracy)
        print(output_string)
        f.write(output_string)

This takes a while which is a perfect chance to go outside to get some fresh air or even go on a hike, depending on how many models you want to run. Let’s take a look what we have got:

Running amazon data set
Best Accuracy : 0.8122
{'vocab_size': 4603, 'num_filters': 64, 'maxlen': 100, 'kernel_size': 5, 'embedding_dim': 50}
Test Accuracy : 0.8457

Running imdb data set
Best Accuracy : 0.8161
{'vocab_size': 4603, 'num_filters': 128, 'maxlen': 100, 'kernel_size': 5, 'embedding_dim': 50}
Test Accuracy : 0.8210

Running yelp data set
Best Accuracy : 0.8127
{'vocab_size': 4603, 'num_filters': 64, 'maxlen': 100, 'kernel_size': 7, 'embedding_dim': 50}
Test Accuracy : 0.8384

Interesting! For some reason the testing accuracy is higher than the training accuracy which might be because there is a large variance in the scores during cross-validation. We can see that we were still not able to break much through the dreaded 80%, which seems to be a natural limit for this data with its given size. Remember that we have a small data set and convolutional neural networks tend to perform the best with large data sets.

Another method for CV is the nested cross-validation (shown here) which is used when the hyperparameters also need to be optimized. This is used because the resulting non-nested CV model has a bias toward the data set which can lead to an overly optimistic score. You see, when doing hyperparameter optimization as we did in the previous example, we are picking the best hyperparameters for that specific training set but this does not mean that these hyperparameters generalize the best.

Conclusion

There you have it: you have learned how to work with text classification with Keras, and we have gone from a bag-of-words model with logistic regression to increasingly more advanced methods leading to convolutional neural networks.

You should be now familiar with word embeddings, why they are useful, and also how to use pretrained word embeddings for your training. You have also learned how to work with neural networks and how to use hyperparameter optimization to squeeze more performance out of your model.

One big topic which we have not covered here left for another time was recurrent neural networks, more specifically LSTM and GRU. Those are other powerful and popular tools to work with sequential data like text or time series. Other interesting developments are currently in neural networks that employ attention which are under active research and seem to be a promising next step since LSTM tend to be heavy on the computation.

You have now an understanding of a crucial cornerstone in natural language processing which you can use for text classification of all sorts. Sentiment analysis is the most prominent example for this, but this includes many other applications such as:

Spam detection in emails
Automatic tagging of texts
Categorization of news articles with predefined topics

You can use this knowledge and the models that you have trained on an advanced project as in this tutorial to employ sentiment analysis on a continuous stream of twitter data with Kibana and Elasticsearch. You could also combine sentiment analysis or text classification with speech recognition like in this handy tutorial using the SpeechRecognition library in Python.