Advanced Python

Read time: 33 minutes (8403 words)

Sooner or later, you will want to create a cool project that will either be useful to you and your work, or will help others. Maybe that project will be written in Python, maybe in some other language. If you choose to write a significant project in Python, there are a few things you should know before getting started.

Publish on GitHub

In today’s development world, your project and code should be visible to the development community on a server like GitHub. There are other places you could publish your work, but GitHub seems to be the first place folks (including those seeking to hire developers) go to find people and code.

Start your project on day one by setting up a GitHub repository, and cloning the repository onto your workstation.

Install Python3

There is still a lot of code out on the Internet that only runs in Python 2. No one is seriously developing new code using that version of Python. You should install and use the latest stable release of Python. Today, that is version 3.7. We covered installing that version at the beginning of the course.

Make sure you also have pip installed as well. Run these two commands to make sure:

$ python --version
Python 2.7.15
$ pip --version
pip 18.1 from /usr/local/lib/python2.7/site-packages/pip (python 2.7)

That is kind of annoying! On my Mac, the default Python is the old one. To use the new version, I need to run these commands:

$ python3 --version
Python 3.6.5
$ pip --version
pip 9.0.3 from /Users/rblack/_acc/_github/mandelbrot/_venv/lib/python3.6/site-packages (python 3.6)

It seems my current setup, on the main Mac I use, only has Python 3.6 installed. That is fine for my work. You do see that the version of pip matches the version of python, and that is important.

Using Pip

Pip installs Python packages, written by other developers, into your local Python installation, in a folder named site-packages. This folder is intended for installing stuff you need locally. Pip knows that, and takes care of getting things properly installed for you.

So far, we have not done anything new. All of this is what we have been doing since we started in this class.

Python VirtualEnv

As you start building large Python projects, you discover that using packages written by others makes it far easier to get your project running. When you make this decision, you are adding a dependency to your project. In order for your project to run, the user needs to install that dependent package you decided to use.

Now, you could include that dependent package in your project, but that raises a problem.

What if there are bugs in that package and the developer fixes them. You might like to ditch the old version and use the new one instead. Unfortunately, that raises still another problem.

What if the new version breaks things in your project? Unfortunately, that sort of thing happens all the time. The solution might be to lock your project down and use one specific version of a package you know works, and do not update it to the new one.

All of this leads to something developers hate. A world where you chase down dependency issues, involving code you did not even write. Some developers call this the “dependency Hell” for obvious reasons.

A solution to this mess was developed by a really cool now ex-Python developer named Ian Bicking. Ian’s tool is called a Python VirtualEnv, a system that creates an isolated “virtual” Python environment in which you build and maintain your code.

Note

You may wonder why Ian is an “ex-Python” developer. He was so good, he was quite in demand for his expertise in things related to Python. He burned out, and wanted to explore more of the software world, and made a switch to working with Javascript! Python’s loss!

Ian’s tool was actually incorporated into python3, and most serious Python developers automatically use this to save themselves a lot of pain later.

Setting up a VirtualEnv

This is not very hard at all, but it requires training yourself to remember that you are using it.

Start off in your project repository and type this:

$ python3 -m venv _venv

Note

That python3 is what I need to type to run the new Python on my Mac. On a PC, just type “python”.

Actually, that last _venv name is what I call my “virtual Environments, you can change that if you wish.

Once this command completes, you will see a new folder in your project named _venv. Inside that folder is a complete mini-python world, which can be kept completely independent of your system Python. Pip is installed, and a separate version of the Python tools you need. (Basically, they are copies of your system Python installation).

Warning

You do NOT want to push this folder to GitHub. Be sure to add this folder name to your .gitignore file to keep it out. (You do have a .gitignore file, right?) Just add a line with “_venv” on it (no quotes).

Activating your VirtualEnv

Just having that folder in your project is not enough. You have to activate it by doing this:

$ _venv\scripts\activate

On a Mac/Linux system use this instead:

$ source _venv/bin/activate

Your command-line prompt should change to look like this:

(_venv) ~/_acc/ $

That is from my Mac. The key is the first part, which tells you that you are now working in your Virtual Environment.

When you type python now (on any system), you will launch the isolated version inside your _venv folder. Pip is also installed.

You can “deactivate” this setup, but I simply close the command-prompt window and everything goes away then. You open up a new command-prompt window, reactivate your VirtualEnv, and go to work. You do need to remember that.

Note

If you work a lot on a Mac or Linux system, there is away to automatically activate a VirtualEnv, but it involves setting up commands on your system. See me if you are interested in that. On a PC, there is another way you can use.

Create a “batch” file in your project folder that looks like this:

workon.bat

_venv\Scripts\activate

Now, just type “workon” and you should be ready to work.

Creating a Requirements File

The next step involved creating a simple text file that lists all of the dependencies you need for your project. For this example, I am going to list Sphinx and the theme I like to use for my documentations. Here is my file:

requirements.txt

sphinx
sphinx-rtd-theme

That name is what folks expect to see, so do not change it.

Now, to install your required packages, do this:

* pip install -r requirements.txt

Most Python developers know that line by heart, and any Python project they clone from GitHub better have that file to make them happy.

Matching specific Packages

Once you run that previous commands, you sill discover that you really needed other packages as well. Pip handles that for you. Here is what I really got when installing this requirements file:

$ pip freeze
alabaster==0.7.12
Babel==2.6.0
certifi==2018.11.29
chardet==3.0.4
docutils==0.14
idna==2.7
imagesize==1.1.0
Jinja2==2.10
MarkupSafe==1.1.0
packaging==18.0
Pygments==2.3.0
pyparsing==2.3.0
pytz==2018.7
requests==2.20.1
six==1.11.0
snowballstemmer==1.2.1
Sphinx==1.8.2
sphinx-rtd-theme==0.4.2
sphinxcontrib-websupport==1.1.0
urllib3==1.24.1

Phew, that is a lot of stuff. All of those were dependencies of the Sphinx project, or the theme I selected. Good thing I did not need to type in all of those commands!

Behind each dependency package name you see the specific version of that package you installed. Version numbers are used by developers to indicate something about their projects. For example, I am using Sphinx version 1.8.2. In most cases, the numbers means this:

• 1 - the major release of this software. A new release does not need to maintain the same user interface.

• 8 - the minor release. This might add new features, but does not alter the user interface

• 2 - the patch level. Usually this number means a bug fix release.

So, I am using Sphinx 1.8.2 on my system. To force pip to install a specific version of a package, you add the “==” and the version number to your requirements.txt file. Leaving that version stuff off, you get the most current release of the package.

Note

Some developers “freeze”” their requirements, by doing this:

$ pip freeze > requirements.txt

This will overwrite your current requirements file with the output we saw earlier. The list will be complete, and each package version you are using will be listed.

Enter PyPi

The Python Packackage Index (PyPi) is a public server where most Python developers put code they feel is ready for others to use.

Note

Some developers may post versions of their project while that project is not really ready for use, and that is fine, as long as that is noted.

PyPi is the server that pip automatically searches to find a package you want to install on your system.

Note

I am not going to cover setting that up, since it is a bit involved. If your project reaches a point where this makes sense, go to the PyPi website, register, and add your project. It is all free, but takes some work to set up.

Create Your Masterpiece!

Once your project is set up, you simple work. All of the packages you install here do not impact other projects on your system. On my Mac at the moment, I have about 50 active projects, most of which I clone from GitHub to study. To make sure I do not zap one of my Python projects while I study a “foreign” piece of code. I set up a VirtualEnv in the new project, use pip to install anything needed by the project based on the project requirements.txt file, and test away. Nothing I do in that project will interfere with any other project on my system.

Neat!