Read time: 39 minutes (9840 words)

We know enough now to build a script that explores a bit of our system.

Note

What is a script? Well, for some reason, developers call small programs they use for a variety of system management tasks scripts, not programs. This lecture was originally part of a course called Python Scripting, which was aimed at folks who manage a lot of computers!

Many systems today come with Python installed (except for Windows). Since Python is so useful for quickly building small programs, most developers automatically make sure they can use Python on their systems. (OK, so some of them use other languages, but Python is way more popular these days!)

Windows folks usually add Python as soon as they fire up a new machine. Since we all have Python installed. Let’s see what we can do with it to help manage our machines.

## What System Am I On?¶

Since Python runs anywhere, it makes sense to think about what needs to be done differently, depending on the system you are using at the moment. Here is how w can do that in Python:

ex0.py
import platform

print(platform.system())
print(platform.processor())
print(platform.machine())
print(platform.python_version())


$python ex0.py Darwin i386 x86_64 3.6.5  The first line tells me I am sitting in front of a Mac. (Why it says Darwin, we need to ask Apple!). It will say Linux or Windows for other systems. The second line is not that useful. I actually have a pretty hot MacBook, and i386 is a generic name for a system running an Intel chip. The following line tells me more. My system is definitely running a 64-bit processor. The last line shows the version of Python I used to run this script. All of these “facts” can be used to help build scripts you can use anywhere. We will use that for our next script. ## Checking the System PATH¶ We can ask Python to access the system PATH variable by doing this: ex1.py import os print(os.environ['PATH'])  $ python ex1.py
/Users/rblack/_acc/_venv/bin:/Library/Frameworks/Python.framework/Versions/3.6/bin:/Users/rblack/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/opt/X11/bin


Phew, what a mess! Well, yes it is, but there is a pattern at work here. If you look closely, each entry is separated from the others by a colon. We can use that fact to break apart what is one long string into a series of short strings. The technique is simple:

Note

If you work on a Windows system, the list is very much longer and Microsoft separates entries using a semicolon. We will use our new insight into detecting the system we are on to deal with this “feature”.

ex2.py
import os
import platform

if platform.system() == "Windows":
sep = ";"
else:
sep = ":"
path = os.environ['PATH']
entries = path.split(sep)
for e in entries:
print(e)


/Users/rblack/_acc/_venv/bin
/Library/Frameworks/Python.framework/Versions/3.6/bin
/Users/rblack/bin
/usr/local/bin
/usr/bin
/bin
/usr/sbin
/sbin
/Library/TeX/texbin
/opt/X11/bin


This list is more useful. Plus, it should work just about anywhere!

## Can We Run a Program?¶

The PATH list you just produced is where your system will look for any program you launch from the command line. Suppose we want to see all the programs in those directories, so we can check if the one we are interested in happens to be listed.

Note

Locating executable programs on a Windows machine is pretty easy, since most of them have names ending in .exe. On other systems, the program names have no extension at all. We might have to deal with that at some point.

In order to find a particular program we are interested in, we need to see what entries are in each directory named in the above list. Fortunately, there is another function in the os library that can do this. I am not going to show the output of this next example (because it is huge), but you can see how it is set up, and how it might be used:

ex3.py
import os
import platform

if platform.system() == "Windows":
sep = ";"
else:
sep = ":"

path = os.environ['PATH']
entries = path.split(sep)
for e in entries:
print(e)
files = os.listdir(e)
for f in files:
print('\t',f)



In this example, the listdir function returns a list of the files in the designated directory. We know the full path to the files (that was what we got earlier), so we can form the full path to the files by concatenating the system path and the file name from this list. If we are looking for the python entry, we might do something like this:

ex4.py
import os
import platform

if platform.system() == "Windows":
sep = ";"
else:
sep = ":"

path = os.environ['PATH']
entries = path.split(sep)
for e in entries:
files = os.listdir(e)
for f in files:
if f == 'python':
fullpath = os.path.join(e,f)
print(fullpath)


$python ex4.py /Users/rblack/_acc/_venv/bin/python /usr/local/bin/python /usr/bin/python  Note Hmmm, it looks like I have several python programs on my system. The system uses the first one it finds when doing a similar search, every time you launch a program. The first entry in this list is because I use an advanced Python development technique that creates a virtual environment for a Python project. That isolates the development tools I use so I can build Python projects for several different versions of Python and use different versions of various libraries I might need for each. We will not explore that idea in this course, but you should explore it later. We used another operating system function here, the os.path.join function which combines parts of a path using the separator convention of the operating system you are using. That means it joins parts with a forward slash on Linux, and back-slash on Windows. There are many more such useful routines in the os library. ## Getting Python to run a program for you¶ We can use another simple feature of Python to get it to run a program for you and collect the output from that program. This example is a bit involved, and we will not get into the details of all of this in this class, For now, watch the action! ex5.py import subprocess p = subprocess.Popen(["python", "--version"],stderr=subprocess.PIPE, stdout=subprocess.PIPE) output = p.communicate() res = output[0] ver = "".join( chr(x) for x in res) print("out:",ver)  $ python ex5.py
out: Python 3.6.5


That looks like something we can use!

To figure out what happened here, we need to know a little more about how programs produce output.

### STDOUT, STDIN, STDERR¶

The Unix world has long had a simple way for programs to communicate with the outside world. They use the idea of pipes connected to standard devices to do simple input and output. The standard input pipe is normally attached to your keyboard for input. The standard output pipe is connected to your screen for output. That last standard pipe is reserved for error messages. It goes by the name standard error. Windows picked up this standard and we can access the output produced by either standard input (stdin) or standard error (stderr). In the above example we caused the system to capture the output from stdin and stderr so we can read them later. The communicate() method returns a tuple with stdin and stderr output, but it captures the output as a stream of bytes, not as a string. That makes using the output difficult.

The last part of this script converts the bytes into a proper string. The line used is called a “comprehension” (just to make your head hurt!) What it does is to scan the list of bytes, and stick each one onto the end of a string. When the smoke clears, we have the string we wanted.

What this entire script did was to start up a separate program from within our running script, let that program do whatever it normally does, and capture whatever output was produced. We got a tuple back, containing the output from the stdin pipe, and the stderr pipe!

I know this will be new to most of you, but it is a valuable thing to know. With it, we can build scripts that run programs for us, capture and process the output, then do other things as well. This can be a powerful way to build complicated scripts! I use this technique to build parts of my course notes. Embedded in my notes are notations that I capture and use to run programs. I collect the output from those programs and put that output back into the notes for you to see. Neat!

#### File redirection¶

Related to this scheme is your operating system’s ability to use redirection when you run programs on the command line. You can use these symbols:

So we could have used this to capture the Python version number:

python --version > version


You will now have a file on your system named version. That file has the string we saw earlier in it!

This technique is often used to automate simple programs and capture the output. We will see more use of these techniques in later scripts.

Suppose your program was not to be found on the system PATH, could we still launch it using this technique?

Sure, just form the full path to the executable file you want to launch. Suppose I need to launch the version of python that is living in /usr/local/bin on my Mac:

ex6.py
import subprocess
import os

pypath = "/usr/local/bin"
pyprog = os.path.join(pypath,"python")

p = subprocess.Popen(
[pyprog, "--version"],
stderr=subprocess.PIPE,
stdout=subprocess.PIPE)
output = p.communicate()
res = output[0]
ver = "".join( chr(x) for x in res)
print("out:",ver)

\$ python ex6.py
out:


It looks like the default version of Python on my Mac is the older version.

## Useful Scripts¶

How you use your system, and what you need to do on that system is a very personal hing. I teach programming, so a lot of the tools I create are designed to help programmers get their work done.

### Finding Program Source Code Files¶

In my C++ classes we build programs with several source code files. These files are not usually sitting in one directory. They may be buried in subdirectories set up to keep all the code about one aspect of a big project organized.

As an example of a script I wrote to help find app the C++ files under some folder, here is one I use all the time:

pyfind.py
import os
import sys

dir = sys.argv[1]
ext = sys.argv[2]
result = []
for root, dirs, files in os.walk(dir):
for name in files:
if name.endswith(ext):
result.append(os.path.join(root,name))

print(" ".join(result));



This script takes two parameters. The first is the path to the top directory I want to search, and the second is the extension string I want to search for. The script prints a list of all file names found, complete with the path to each file. This list is separated by spaces, since I use the list as part of a building system you will learn about in a later course.

Here is what it looks like running:




### Generating Images for Lectures¶

Here is a test script I use to generate image files containing nice figures for use in my lecture notes. The tool I use to produce these images is called TikZ, with depends on the LaTeX program to generate a PDf file Another program converst the PDF file into an image file I can use in my lectures.

To build my image file, I need to write a short piece of code in the TikZ language. To keep that code easy to write, I strip away all of the surrounding text needed t build a PDF file, leaving something that looks like this:

sample.fig
\draw[style=dashed] (2,.5) circle (0.5);
\draw[fill=green!50] (1,1)
ellipse (.5 and 1);
\draw[fill=blue] (0,0) rectangle (1,1);
\draw[style=thick]
(3,.5) -- +(30:1) arc(30:80:1) -- cycle;


To get this file ready for LaTeX processing, I need to surround my code text in a standard pile of text that never changes. So, I set up this file as well:

base.tex
\documentclass[preview]{standalone}
\usepackage{tikz}
\begin{document}
\begin{tikzpicture}
%s
\end{tikzpicture}
\end{document}


What I need now is a simple bit of Python to put these two files together:

fig_generator.py
base = open('base.tex').read()
out = base % fig
fout = open('figure.tex', 'w')
fout.write(out)
fout.close()



Here is the combined file, ready for processing by LaTeX:

figure.tex
\documentclass[preview]{standalone}
\usepackage{tikz}
\begin{document}
\begin{tikzpicture}
\draw[style=dashed] (2,.5) circle (0.5);
\draw[fill=green!50] (1,1)
ellipse (.5 and 1);
\draw[fill=blue] (0,0) rectangle (1,1);
\draw[style=thick]
(3,.5) -- +(30:1) arc(30:80:1) -- cycle;

\end{tikzpicture}
\end{document}


The output from LaTeX is messy, so I am not showing it here.

Note

LaTeX is a powerful tool freely avialable on all systems, and one oyu may need to larn about if you go to graduate school. Many schools want a Thesis or Dissertation to be processed by LaTeX so the result will end up a nice book!

To test the code needed to build the image. I use this scrtipt:

genimage.py
import os

status = os.system('python fig_generator.py')
assert status == 0
status = os.system('pdflatex --interaction=nonstop figure')
assert status == 0
status = os.system('convert -density 300 figure.pdf -quality 90 figure.png')
assert status == 0


I do not need to capture the output from these commands. So, I can use another tool in the Python os library to just run the program.

Those assert lines are a nice way tomake syre the program actually worked before trying the next step. If the assert line failes to produce a True`, the program will halt with an error message.

And, finally, here is the image produced:

That is a lot nicer than fighting with some graphics tool and trying to build something like this!

Python can be your best friend as a developer.