Python Packages

Read time: 28 minutes (7198 words)

Last time we looked at Python’s Modules. Basically, a module is a simple way to break up a program that is getting too big. We cut the big file up into chunks, hopefully containing related parts of the program, and place them in a separate file.

That really helps keep yu focused on pne part o f the program at a time. As the program grows in size, you will appreciate this system a lot.

Too Many Modules

A program can get even bigger, and the module idea starts to fall apart. Sure, you can create hundreds of files in a single folder, but do you really want to wade through a giant list of files looking for the one you need to look at?

We can take the module idea one step further. We can group modules into a separate directory. With a little tuning, we will call this new directory a Python Package.

Package Structure

You will find many Python programmers using a simple scheme to set up a package. You simply move your modules into the new folder, then add a file named __init__.py. Often that file has no text in it, but in older versions of Python, that file needed to be in a folder for Python to accept it as a pacKage!

Modern Python versions (3.3+) do not need that __init__.py file, but it is often put in a package in case it is needed later (More on that soon in this discussion!)

Let’s create a simple example.

Package Setup

In this example, I will show command line steps to set up a simple Python Project. We will see a new way to build a program as we do this.

Note

Of course, you can create this setup using a tool like the Windows File Explorer on a PC, or the Finder tool on a Mac.

Pick a Project Name

Picking a name is an important first step. Developers looking for programs (maybe on GitHub) to help them with something will search for information based on some name, so your name should be helpful to those folks.

For our simple example, we will use UnitConverter as our project name:

$ mkdir UnitConverter
$ cd UnitConverter

Create Basic Folders

We willl organize our project in a few basic folders:

• source - where the Python code will live

• tests - where testing code will live (see below)

• docs - where documentation will live

Here is how we can set this part up:

> mkdir source
> mkdir tests
> mkdir docs

Ready to Code?

Let’s start coding.

In the source folder, create this file:

source/temps.py

# temperature conversions

def ftoc(value):
    return (value - 32) * 5 / 9

def ctof(value):
    return (value * 9 / 5) + 32

if __name__ == '__main__':
    print("ftoc(32)", ftoc(32))
    print("ctof(0)", ctof(0))

Notice that last if block. This is a python module, but we are going to test it directly.

Now do this:

$ python -m temps
ftoc(32) 0.0
ctof(0) 32.0

Let’s run this as a module. To do that, we need a simple application:

source/units.py

# Unit Converter program
from temps import ftoc, ctof

def main():
    print("ftoc(32)", ftoc(32))
    print("ctof(100)", ctof(100))

if __name__ == '__main__':
    main()

Now, run this application:

$ python units.py
ftoc(32) 0.0
ctof(100) 212.0

It looks like this works as expected. Notice that I used that standard if block at the end. This is a common practice in the Python world.

Package Time

Now, create a new folder under source:

> mkdir units

Create an empty file named __init__.py in this new folder. (You can just open up gVim and save the initially empty file with the name needed.)

Now, move your temps.py into this folder and modify the application so it looks like this:

source/units.py

# Unit Converter program
from units.temps import ftoc, ctof

def main():
    print("ftoc(32)", ftoc(32))
    print("ctof(100)", ctof(100))

if __name__ == '__main__':
    main()

Now, run this modified application:

$ python units.py
ftoc(32) 0.0
ctof(100) 212.0

This works, but it has an ugly import line. We created a package we named units, but we needed to know the name of the python module inside of that package. Users do not need to see that, or even worry about that. So, how do we fix this?

Remember how we used the sqrt function:

All we did was something like this:

from math import sqrt
...
x = sqrt(2)

Or, we could do this:

import math

x = math.sqrt(3)

Namespaces

Python maintains a list of all names defined in your code at the moment. This list has a bunch of names Python uses internally, and that list expands as your define things in your code.

There is a built-in function named dir that you can use to see what names are defined. Let’s modify our application so we can see this list:

In the first case, we are making the name sqrt directly part of our local namespace, a fancy term meaning all the names of things defined in our code at the moment.

In the second case, sqrt is not in our namespace, but ``math is. We know that sqrt is in that math module )or is os a package?) So we tell Python to access the needed function with a slightly longer call.

Note

Best practices in programming tells us to NEVER use from <module import * because this “pollutes” our programs namespace. YMMV!

We would like for our package users to just be able to use our cool functions directly, and not even know ow we have organized our code.

Examining Your Namespace

Fire up the Python interpreter in the source directory and do this:

>>> dir()
['__annotations__', '__builtins__', '__doc__', '__loader__', '__name__', '__package__', '__spec__']

Nothins looks familiar there. These are all default names created by Python.

Next, import our package:

>>> import units
>>> dir()
['__annotations__', '__builtins__', '__doc__', '__loader__', '__name__', '__package__', '__spec__', 'units']

We still do not have access to the functions. Try again:

>>> from units import *
>>> dir()
['__annotations__', '__builtins__', '__doc__', '__loader__', '__name__', '__package__', '__spec__', 'units']

That did not work. One more try:

>>> from units.temps import *
>>> dir()
['__annotations__', '__builtins__', '__doc__', '__loader__', '__name__', '__package__', '__spec__', 'ctof', 'ftoc', 'units']

Now, we see our two functions.

Initializing the Namespace

What we want to do is to get Python to add these names to the namespace using the first simple import option.

That is where the __init__.py file comes in!

This file is processed by Python when you import the package. We can add any code here we like, but it is common to keep “real” code out of here for now.

Modify your _init__.py file (in the units folder) so it looks like this:

source/units/__init__.py

from units.temps import *

Check that we can now access our functions:

>>> import units
>>> dir(units)
['__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', 'ctof', 'ftoc', 'temps']

Success!

It seems odd to need to reference the package name inside of the package. That makes it hard to “reuse” this package in another project where the package name might need to be changed. Python lets us simplify the import line so this problem does not exist. Simply drop the package name from the import line, leaving the dot and the module name:

source/units/__init__.py

from .temps import *

Now, our application can look like this:

source/units.py

# Unit Converter program

from units import ftoc, ctof

def main():
    print("ftoc(32)", ftoc(32))
    print("ctof(100)", ctof(100))

if __name__ == '__main__':
    main()

Try it out:

$ python units.py
ftoc(32) 0.0
ctof(100) 212.0

Now we have a working package. We can add more unit conversions!

Add Another Module

We can add anothe rmodule to our 11units`` package as easliy as dropping another file in the folder, and adding a line to the __init__.py file:

source/units/length.py

# length conversions

FTOM = 0.3048

def ftom(val):
    return val * FTOM

def mtof(val):
    return val / FTOM

if __name__ == '__main__':
    print("ftom(1)", ftom(1))
    print("mtof(1)", mtof(1))

source/units/__init__.py

from .temps import *
from .lengths import *

We need to add more calls to out test application. Hmmm, let’s rename it to test_units.py

source/test_units.py

# Unit Converter program

def show_names(mname):
    print(dir(mname))

from units import *

def main():
    print("ftoc(32)", ftoc(32))
    print("ctof(100)", ctof(100))
    print("ftom(1)", ftom(1))
    print("mtof(1)", mtof(1))

if __name__ == '__main__':
    main()

Try it out:

$ python test_units.py
ftoc(32) 0.0
ctof(100) 212.0
ftom(1) 0.3048
mtof(1) 3.280839895013123

Wrapping Up

We started off by building an application suitable for public use. We need a real front end interface to our new package to make is “user friendly”. Try this in the source directory:

source/run.py

import sys
from units import *

def usage():
    print("Unit Converter (v0.1)")
    print("Usage: python -m units [option] value")
    print("\toptions:")
    print("\t\t-ctof : degrees celcius to farenheit")
    print("\t\t-ftoc : farenheit to degrees celcius")
    print("\t\t-dtor : degrees to radians")
    print("\t\t-rtod : radians to degrees")
    print("\t\t-ftom : feet to meters")
    print("\t\t-mtof : meters to feet")
    sys.exit()

def run():
    # check for proper use
    if len(sys.argv) != 3: usage()
    
    # process option
    option = sys.argv[1][1:]
    value = float(sys.argv[2])
    
    # process request
    if option   == 'ctof':
        print(ctof(value))
    elif option == 'ftoc':
        print(ftoc(value))
    elif option == 'dtor':
        print(dtor(value))
    elif option == 'rtod':
        print(rtod(value))
    elif option == 'ftom':
        print(ftom(value))
    elif option == 'mtof':
        print(mtof(value))
    else:
        print("unknown conversion")

def tests():
    pass

if __name__ == '__main__':
    run()

Try it out, in the source folder:

$ python run.py
Unit Converter (v0.1)
Usage: python -m units [option] value
	options:
		-ctof : degrees celcius to farenheit
		-ftoc : farenheit to degrees celcius
		-dtor : degrees to radians
		-rtod : radians to degrees
		-ftom : feet to meters
		-mtof : meters to feet

$ python run.py -ftom 5280
1609.344

I will leave it to you to complete the unit conversion package.

Testing

Why did we change the name of our old test program to test_units?

It is vitally important that you test your code. That is why we set up a tests folder for this project. Move the test_units.py file into tests. This is where we will put all test code for this project.

But how do we actually test?

The code we wrote originally was just a test application that does test the package code, but this is a mess to set up for a large project. Python developers fixed that problem long ago by creating a cool package (!) called PyTest.

Python has literally hundreds (thousands) of cool packages around. We will focus on testing here.

Assuming you have pip installed, which should be true if you installed Python from the official website, do this from a command line:

$ pip --version
pip 20.2.4 from /Users/rblack/_projects/lectures/cosc1336/_venv/lib/python3.8/site-packages/pip (python 3.8)

That proves pip is installed.

Pip is the standard tool for loading publicly available Python packages from the Python Package Index (PyPi), where open source Python developers publish their code. We will try out PyTest for testing our package.

$ pip install pytest

Warning

You need to be connected to the Internet when you run this command. The output will show you messages as all the needed files are downloaded and installed on your system.

Ditch that original test code, and replace it with this version:

tests/test_units.py

# Unit Converter tests

from source.units import *

def test_ftoc():
    assert(ftoc(32) == 0.0)

def test_ctof():
    assert(ctof(0) == 32.0)

Once that file is on place, run the test command from the command line at the top of your project folder:

System Message: ERROR/6 (/Users/rblack/_projects/lectures/cosc1336/modules-packages/02-packages.rst, line 483)

Command 'pytest tests' failed: [Errno 2] No such file or directory: 'pytest'

Hey, you are well on your way to becoming a “real” Python developer!