Exploring your system
#####################

..  wordcount::
..  include::   /references.inc
..  vim:filetype=rst spell:

We know enough now to build a script that explores a bit of our system. 

..  note::

    What is a ``script``? Well, for some reason, developers call small programs
    they use for a variety of system management tasks ``scripts``, not
    programs. This lecture was originally part of a course called ``Python
    Scripting``, which was aimed at folks who manage a lot of computers!


Many systems today come with Python installed (except for Windows). Since
Python is so useful for quickly building small programs, most developers
automatically make sure they can use Python on their systems. (OK, so some of
them use other languages, but Python is way more popular these days!)

Windows folks usually add Python as soon as they fire up a new machine.  Since
we all have Python installed. Let's see what we can do with it to help manage
our machines.

What System Am I On?
********************

Since Python runs anywhere, it makes sense to think about what needs to be done
differently, depending on the system you are using at the moment. Here is how
w can do that in Python:

..  literalinclude::   code/ex0.py
    :caption: ex0.py

..  command-output::    python ex0.py
    :cwd: code
   
The first line tells me I am sitting in front of a Mac. (Why it says
``Darwin``, we need to ask Apple!). It will say ``Linux`` or ``Windows`` for
other systems.

The second line is not that useful. I actually have a pretty hot MacBook, and
``i386`` is a generic name for a system running an Intel chip. The following
line tells me more.  My system is definitely running a 64-bit processor.

The last line shows the version of Python I used to run this script.

All of these "facts" can be used to help build scripts you can use anywhere. We
will use that for our next script.

Checking the System PATH
************************

We can ask Python_ to access the system ``PATH`` variable by doing this:

..  literalinclude::    code/ex1.py
    :caption: ex1.py


..  command-output::    python ex1.py
    :cwd: code

Phew, what a mess! Well, yes it is, but there is a pattern at work here. If you
look closely, each entry is separated from the others by a colon. We can
use that fact to break apart what is one long string into a series of short strings. The
technique is simple:

..  note::

    If you work on a Windows system, the list is very much longer and Microsoft
    separates entries using a semicolon. We will use our new insight into
    detecting the system we are on to deal with this "feature".
    

..  literalinclude::    code/ex2.py
    :caption: ex2.py

..  program-output::    python ex2.py
    :cwd: code


This list is more useful. Plus, it should work just about anywhere!

Can We Run a Program?
*********************

The **PATH** list you just produced is where your system will look for any
program you launch from the command line. Suppose we want to see all the
programs in those directories, so we can check if the one we are interested in
happens to be listed.

..  note::

    Locating executable programs on a Windows machine is pretty easy, since
    most of them have names ending in **.exe**. On other systems, the program
    names have no extension at all. We might have to deal with that at some
    point.

In order to find a particular program we are interested in, we need to see what
entries are in each directory named in the above list. Fortunately, there is
another function in the ``os`` library that can do this. I am not going to show
the output of this next example (because it is huge), but you can see how it is
set up, and how it might be used:

..  literalinclude::    code/ex3.py
    :caption: ex3.py

In this example, the ``listdir`` function returns a list of the files in the
designated directory. We know the full path to the files (that was what we got
earlier), so we can form the full path to the files by concatenating the system
path and the file name from this list. If we are looking for the ``python``
entry, we might do something like this:

..  literalinclude::    code/ex4.py
    :caption: ex4.py

..  command-output::    python ex4.py
    :cwd: code

..  note::

    Hmmm, it looks like I have several ``python`` programs on my system. The
    system uses the first one it finds when doing a similar search, every time
    you launch a program. The first entry in this list is because I use an
    advanced Python development technique that creates a ``virtual environment``
    for a Python project. That isolates the development tools I use so I can
    build Python projects for several different versions of Python and use
    different versions of various libraries I might need for each. We will not
    explore that idea in this course, but you should explore it later.

We used another operating system function here, the ``os.path.join`` function
which combines parts of a path using the separator convention of the operating
system you are using. That means it joins parts with a forward slash on Linux,
and back-slash on Windows. There are many more such useful routines in the
``os`` library.

Getting Python_ to run a program for you
****************************************

We can use another simple feature of Python_ to get it to run a program for you
and collect the output from that program. This example is a bit involved, and
we will not get into the details of all of this in this class, For now, watch
the action!

..  literalinclude::    code/ex5.py
    :caption: ex5.py

..  command-output::    python ex5.py
    :cwd: code

That looks like something we can use!

To figure out what happened here, we need to know a little more about how
programs produce output.

STDOUT, STDIN, STDERR
=====================

The Unix world has long had a simple way for programs to communicate with the
outside world. They use the idea of ``pipes`` connected to standard devices to
do simple input and output. The ``standard input`` pipe is normally attached to
your keyboard for input. The ``standard output`` pipe is connected to your
screen for output. That last standard pipe is reserved for error messages. It
goes by the name ``standard error``. Windows picked up this standard and we can
access the output produced by either ``standard input`` (``stdin``) or
``standard error`` (``stderr``). In the above example we caused the system to
capture the output from ``stdin`` and ``stderr`` so we can read them later.
The ``communicate()`` method returns a tuple with ``stdin`` and ``stderr``
output, but it captures the output as a stream of bytes, not as a string. That
makes using the output difficult.

The last part of this script converts the bytes into a proper string. The line
used is called a "comprehension" (just to make your head hurt!) What it does is
to scan the list of bytes, and stick each one onto the end of a string. When the
smoke clears, we have the string we wanted.

What this entire script did was to start up a separate program from
within our running script, let that program do whatever it normally does, and
capture whatever output was produced. We got a tuple back, containing the output
from the ``stdin`` pipe, and the ``stderr`` pipe!

I know this will be new to most of you, but it is a valuable thing to know.
With it, we can build scripts that run programs for us, capture and process the
output, then do other things as well. This can be a powerful way to build
complicated scripts! I use this technique to build parts of my course notes.
Embedded in my notes are notations that I capture and use to run programs. I
collect the output from those programs and put that output back into the notes
for you to see. Neat!

File redirection
----------------

Related to this scheme is your operating system's ability to use
``redirection`` when you run programs on the ``command line``. You can use
these symbols:

..  csv-table

    "<", use following file name as input
    ">", use following filename to capture output
    "2>" use following file name to capture error output

So we could have used this to capture the Python version number:

..  code-block:: text

    python --version > version

You will now have a file on your system named ``version``. That file has the
string we saw earlier in it!

This technique is often used to automate simple programs and capture the
output. We will see more use of these techniques in later scripts.

Running your program
====================

Suppose your program was not to be found on the system ``PATH``, could we still
launch it using this technique?

Sure, just form the full path to the executable file you want to launch.
Suppose I need to launch the version of ``python`` that is living in ``/usr/local/bin`` on my Mac:

..  literalinclude::    code/ex6.py
    :caption: ex6.py

..  command-output::    python ex6.py
    :cwd: code


It looks like the default version of Python on my Mac is the older version. 

Useful Scripts
**************

How you use your system, and what you need to do on that system is a very
personal hing. I teach programming, so a lot of the tools I create are designed
to help programmers get their work done.

Finding Program Source Code Files
=================================

In my C++ classes we build programs with several source code files. These files
are not usually sitting in one directory. They may be buried in subdirectories
set up to keep all the code about one aspect of a big project organized.

As an example of a script I wrote to help find app the C++ files under some
folder, here is one I use all the time:

..  literalinclude::    code/pyfind.py
    :caption: pyfind.py

This script takes two parameters. The first is the path to the top directory I
want to search, and the second is the extension string I want to search for.
The script prints a list of all file names found, complete with the path to
each file. This list is separated by spaces, since I use the list as part of a
building system you will learn about in a later course.

Here is what it looks like running:

..  program-output::    python pyfind.py /Users/rblack/_acc/_github/Mandelbrot3/lib .cpp
    :cwd: code

Generating Images for Lectures
==============================

Here is a test script I use to generate image files containing nice figures for
use in my lecture notes. The tool I use to produce these images is called TikZ,
with depends on the LaTeX program to generate a PDf file Another program
converst the PDF file into an image file I can use in my lectures. 

To build my image file, I need to write a short piece of code in the TikZ_ language. To keep that code easy to write, I strip away all of the surrounding text needed t build a PDF file, leaving something that looks like this:

..  literalinclude::    code/sample.fig
    :caption: sample.fig

To get this file ready for LaTeX_ processing, I need to surround my code text in a standard pile of text that never changes. So, I set up this file as well:

..  literalinclude::    code/base.tex
    :caption: base.tex

What I need now is a simple bit of Python to put these two files together:

..  literalinclude::    code/fig_generator.py
    :caption: fig_generator.py

Here is the combined file, ready for processing by LaTeX_:

..  literalinclude::    code/figure.tex
    :caption: figure.tex


The output from LaTeX_  is messy, so I am not showing it here.

..  note::

    LaTeX_ is a powerful tool freely avialable on all systems, and one oyu may
    need to larn about if you go to graduate school. Many schools want a Thesis
    or Dissertation to be processed by LaTeX_ so the result will end up a nice
    book!

To test the code needed to build the image. I use this scrtipt:

..  literalinclude::    code/genimage.py
    :caption: genimage.py

I do not need to capture the output from these commands. So, I can use another tool
in the Python ``os`` library to just run the program.

Those ``assert`` lines are a nice way tomake syre the program actually worked
before trying the next step. If the ``assert`` line failes to produce a
`True``, the program will halt with an error message.

And, finally, here is the image produced:

..  image:: code/figure.png
    :align: center


That is a lot nicer than fighting with some graphics tool and trying to build something like this!

Python can be your best friend as a developer.