Advanced Python
###############

..  include::	/references.inc
..  wordcount::
..  vim:filetype=rst spell:


Sooner or later, you will want to create a cool project that will either be
useful to you and your work, or will help others. Maybe that project will be
written in Python, maybe in some other language. If you choose to write a
significant project in Python, there are a few things you should know before
getting started.

Publish on GitHub
*****************

In today's development world, your project and code should be visible to the
development community on a server like GitHub_. There are other places you could
publish your work, but GitHub_ seems to be the first place folks (including
those seeking to hire developers) go to find people and code.

Start your project on day one by setting up a GitHub_ repository, and cloning
the repository onto your workstation.

Install Python3
***************

There is still a lot of code out on the Internet that only runs in Python 2. No
one is seriously developing new code using that version of Python. You should
install and use the latest stable release of Python. Today, that is version 3.7.
We covered installing that version at the beginning of the course.

Make sure you also have ``pip`` installed as well. Run these two commands to make
sure:

..	code-block:: bash

   $ python --version
   Python 2.7.15
   $ pip --version
   pip 18.1 from /usr/local/lib/python2.7/site-packages/pip (python 2.7)

That is kind of annoying! On my Mac, the default Python is the old one. To use
the new version, I need to run these commands:

..  code-block:: bash

   $ python3 --version
   Python 3.6.5
   $ pip --version
   pip 9.0.3 from /Users/rblack/_acc/_github/mandelbrot/_venv/lib/python3.6/site-packages (python 3.6)

It seems my current setup, on the main Mac I use, only has Python 3.6
installed. That is fine for my work. You do see that the version of ``pip``
matches the version of ``python``, and that is important.

Using Pip
*********

``Pip`` installs Python packages, written by other developers, into your local
Python installation, in a folder named ``site-packages``. This folder is
intended for installing stuff you need locally. Pip knows that, and takes care
of getting things properly installed for you.

So far, we have not done anything new. All of this is what we have been doing
since we started in this class.

Python VirtualEnv
*****************

As you start building large Python projects, you discover that using packages
written by others makes it far easier to get your project running. When you
make this decision, you are adding a ``dependency`` to your project. In order
for your project to run, the user needs to install that dependent package you
decided to use.

Now, you could include that dependent package in your project, but that raises a
problem. 

What if there are bugs in that package and the developer fixes them. You might
like to ditch the old version and use the new one instead. Unfortunately, that
raises still another problem.

What if the new version breaks things in your project? Unfortunately, that sort
of thing happens all the time. The solution might be to lock your project down
and use one specific version of a package you know works, and do not update it
to the new one.

All of this leads to something developers hate. A world where you chase down
dependency issues, involving code you did not even write. Some developers call
this the "dependency Hell" for obvious reasons.

A solution to this mess was developed by a really cool now ex-Python developer
named `Ian Bicking`_. Ian's tool is called a Python VirtualEnv, a system that
creates an isolated "virtual" Python environment in which you build and
maintain your code. 

..  note::

    You may wonder why Ian is an "ex-Python" developer. He was so good, he was
    quite in demand for his expertise in things related to Python. He burned
    out, and wanted to explore more of the software world, and made a switch to
    working with Javascript! Python's loss!

Ian's tool was actually incorporated into python3, and most serious Python
developers automatically use this to save themselves a lot of pain later. 

Setting up a VirtualEnv
***********************

This is not very hard at all, but it requires training yourself to remember
that you are using it.

Start off in your project repository and type this:

..  code-block:: bash

    $ python3 -m venv _venv

..  note::

    That ``python3`` is what I need to type to run the new Python on my Mac. On
    a PC, just type "python".

Actually, that last ``_venv`` name is what I call my "virtual Environments, you
can change that if you wish.

Once this command completes, you will see a new folder in your project named
``_venv``. Inside that folder is a complete mini-python world, which can be
kept completely independent of your system Python. Pip is installed, and a
separate version of the Python tools you need. (Basically, they are copies of
your system Python installation).

..  warning::

    You do NOT want to push this folder to GitHub_. Be sure to add this folder
    name to your ``.gitignore`` file to keep it out. (You do have a
    ``.gitignore`` file, right?) Just add a line with "_venv" on it (no quotes).

Activating your VirtualEnv
===========================

Just having that folder in your project is not enough. You have to activate it
by doing this:

..  code-block:: bash

    $ _venv\scripts\activate

On a Mac/Linux system use this instead:

..  code-block:: bash

    $ source _venv/bin/activate

Your command-line prompt should change to look like this:

..  code-block:: bash

    (_venv) ~/_acc/ $

That is from my Mac. The key is the first part, which tells you that you are
now working in your ``Virtual Environment``.

When you type ``python`` now (on any system), you will launch the isolated
version inside your ``_venv`` folder. ``Pip`` is also installed.

You can "deactivate" this setup, but I simply close the command-prompt window
and everything goes away then. You open up a new command-prompt window,
reactivate your VirtualEnv, and go to work. You do need to remember that.

..  note::

    If you work a lot on a Mac or Linux system, there is away to automatically
    activate a VirtualEnv, but it involves setting up commands on your system.
    See me if you are interested in that. On a PC, there is another way you
    can use.

    Create a "batch" file in your project folder that looks like this:

    ..  literalinclude::    code/workon.bat
        :caption: workon.bat

    Now, just type "workon" and you should be ready to work.

    
Creating a Requirements File
============================

The next step involved creating a simple text file that lists all of the
dependencies you need for your project. For this example, I am going to list
Sphinx_ and the theme I like to use for my documentations. Here is my file:

..  literalinclude::    code/requirements.txt
    :caption: requirements.txt

That name is what folks expect to see, so do not change it.

Now, to install your required packages, do this:

..  code-block:: bash

    * pip install -r requirements.txt

Most Python developers know that line by heart, and any Python project they clone from
GitHub_ better have that file to make them happy.

Matching specific Packages
==========================

Once you run that previous commands, you sill discover that you really needed
other packages as well. Pip handles that for you. Here is what I really got
when installing this requirements file:

..  code-block:: bash

    $ pip freeze
    alabaster==0.7.12
    Babel==2.6.0
    certifi==2018.11.29
    chardet==3.0.4
    docutils==0.14
    idna==2.7
    imagesize==1.1.0
    Jinja2==2.10
    MarkupSafe==1.1.0
    packaging==18.0
    Pygments==2.3.0
    pyparsing==2.3.0
    pytz==2018.7
    requests==2.20.1
    six==1.11.0
    snowballstemmer==1.2.1
    Sphinx==1.8.2
    sphinx-rtd-theme==0.4.2
    sphinxcontrib-websupport==1.1.0
    urllib3==1.24.1

Phew, that is a lot of stuff. All of those were dependencies of the Sphinx_
project, or the theme I selected. Good thing I did not need to type in all of
those commands!

Behind each dependency package name you see the specific version of that
package you installed. Version numbers are used by developers to indicate
something about their projects. For example, I am using Sphinx_ version 1.8.2.
In most cases, the numbers means this:

    * 1 - the major release of this software. A new release does not need to
      maintain the same user interface.

    * 8 - the minor release. This might add new features, but does not alter
      the user interface

    * 2 - the patch level. Usually this number means a bug fix release.

So, I am using Sphinx_ 1.8.2 on my system. To force ``pip`` to install a
specific version of a package, you add the "==" and the version number to your
``requirements.txt`` file. Leaving that version stuff off, you get the most
current release of the package.

..  note::

    Some developers "freeze"" their requirements, by doing this:

    ..  code-block:: bash

        $ pip freeze > requirements.txt

    This will overwrite your current requirements file with the output we saw
    earlier. The list will be complete, and each package version you are using
    will be listed.

Enter PyPi
==========

The ``Python Packackage Index (PyPi)`` is a public server where most Python
developers put code they feel is ready for others to use. 

..	note::

    Some developers may post versions of their project while that project is
    not really ready for use, and that is fine, as long as that is noted.

PyPi_ is the server that ``pip`` automatically searches to find a package you
want to install on your system.

..  note::

    I am not going to cover setting that up, since it is a bit involved. If
    your project reaches a point where this makes sense, go to the PyPi_
    website, register, and add your project. It is all free, but takes some
    work to set up.

Create Your Masterpiece!
************************

Once your project is set up, you simple work. All of the packages you install
here do not impact other projects on your system. On my Mac at the moment, I
have about 50 active projects, most of which I clone from GitHub_ to study. To
make sure I do not zap one of my Python projects while I study a "foreign"
piece of code. I set up a VirtualEnv in the new project, use ``pip`` to install
anything needed by the project based on the project ``requirements.txt`` file,
and test away. Nothing I do in that project will interfere with any other
project on my system.

Neat!