Exploring your system¶
Read time: 39 minutes (9840 words)
We know enough now to build a script that explores a bit of our system.
Note
What is a script? Well, for some reason, developers call small programs
they use for a variety of system management tasks scripts, not
programs. This lecture was originally part of a course called Python
Scripting, which was aimed at folks who manage a lot of computers!
Many systems today come with Python installed (except for Windows). Since Python is so useful for quickly building small programs, most developers automatically make sure they can use Python on their systems. (OK, so some of them use other languages, but Python is way more popular these days!)
Windows folks usually add Python as soon as they fire up a new machine. Since we all have Python installed. Let’s see what we can do with it to help manage our machines.
What System Am I On?¶
Since Python runs anywhere, it makes sense to think about what needs to be done differently, depending on the system you are using at the moment. Here is how w can do that in Python:
import platform
print(platform.system())
print(platform.processor())
print(platform.machine())
print(platform.python_version())
$ python ex0.py
Darwin
i386
x86_64
3.8.6
The first line tells me I am sitting in front of a Mac. (Why it says
Darwin, we need to ask Apple!). It will say Linux or Windows for
other systems.
The second line is not that useful. I actually have a pretty hot MacBook, and
i386 is a generic name for a system running an Intel chip. The following
line tells me more. My system is definitely running a 64-bit processor.
The last line shows the version of Python I used to run this script.
All of these “facts” can be used to help build scripts you can use anywhere. We will use that for our next script.
Checking the System PATH¶
We can ask Python to access the system PATH variable by doing this:
import os
print(os.environ['PATH'])
$ python ex1.py
/Users/rblack/_projects/lectures/cosc1336/_venv/bin:/usr/local/sbin:/Users/rblack/.poetry/bin:/Users/rblack/bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/Library/Apple/usr/bin:/Users/rblack/go/bin
Phew, what a mess! Well, yes it is, but there is a pattern at work here. If you look closely, each entry is separated from the others by a colon. We can use that fact to break apart what is one long string into a series of short strings. The technique is simple:
Note
If you work on a Windows system, the list is very much longer and Microsoft separates entries using a semicolon. We will use our new insight into detecting the system we are on to deal with this “feature”.
import os
import platform
if platform.system() == "Windows":
sep = ";"
else:
sep = ":"
path = os.environ['PATH']
entries = path.split(sep)
for e in entries:
print(e)
/Users/rblack/_projects/lectures/cosc1336/_venv/bin
/usr/local/sbin
/Users/rblack/.poetry/bin
/Users/rblack/bin
/usr/local/sbin
/usr/local/bin
/usr/bin
/bin
/usr/sbin
/sbin
/Library/TeX/texbin
/Library/Apple/usr/bin
/Users/rblack/go/bin
This list is more useful. Plus, it should work just about anywhere!
Can We Run a Program?¶
The PATH list you just produced is where your system will look for any program you launch from the command line. Suppose we want to see all the programs in those directories, so we can check if the one we are interested in happens to be listed.
Note
Locating executable programs on a Windows machine is pretty easy, since most of them have names ending in .exe. On other systems, the program names have no extension at all. We might have to deal with that at some point.
In order to find a particular program we are interested in, we need to see what
entries are in each directory named in the above list. Fortunately, there is
another function in the os library that can do this. I am not going to show
the output of this next example (because it is huge), but you can see how it is
set up, and how it might be used:
import os
import platform
if platform.system() == "Windows":
sep = ";"
else:
sep = ":"
path = os.environ['PATH']
entries = path.split(sep)
for e in entries:
print(e)
files = os.listdir(e)
for f in files:
print('\t',f)
In this example, the listdir function returns a list of the files in the
designated directory. We know the full path to the files (that was what we got
earlier), so we can form the full path to the files by concatenating the system
path and the file name from this list. If we are looking for the python
entry, we might do something like this:
import os
import platform
if platform.system() == "Windows":
sep = ";"
else:
sep = ":"
path = os.environ['PATH']
entries = path.split(sep)
for e in entries:
files = os.listdir(e)
for f in files:
if f == 'python':
fullpath = os.path.join(e,f)
print(fullpath)
$ python ex4.py
/Users/rblack/_projects/lectures/cosc1336/_venv/bin/python
Traceback (most recent call last):
File "ex4.py", line 12, in <module>
files = os.listdir(e)
FileNotFoundError: [Errno 2] No such file or directory: '/Users/rblack/bin'
Note
Hmmm, it looks like I have several python programs on my system. The
system uses the first one it finds when doing a similar search, every time
you launch a program. The first entry in this list is because I use an
advanced Python development technique that creates a virtual environment
for a Python project. That isolates the development tools I use so I can
build Python projects for several different versions of Python and use
different versions of various libraries I might need for each. We will not
explore that idea in this course, but you should explore it later.
We used another operating system function here, the os.path.join function
which combines parts of a path using the separator convention of the operating
system you are using. That means it joins parts with a forward slash on Linux,
and back-slash on Windows. There are many more such useful routines in the
os library.
Getting Python to run a program for you¶
We can use another simple feature of Python to get it to run a program for you and collect the output from that program. This example is a bit involved, and we will not get into the details of all of this in this class, For now, watch the action!
import subprocess
p = subprocess.Popen(["python", "--version"],stderr=subprocess.PIPE, stdout=subprocess.PIPE)
output = p.communicate()
res = output[0]
ver = "".join( chr(x) for x in res)
print("out:",ver)
$ python ex5.py
out: Python 3.8.6
That looks like something we can use!
To figure out what happened here, we need to know a little more about how programs produce output.
STDOUT, STDIN, STDERR¶
The Unix world has long had a simple way for programs to communicate with the
outside world. They use the idea of pipes connected to standard devices to
do simple input and output. The standard input pipe is normally attached to
your keyboard for input. The standard output pipe is connected to your
screen for output. That last standard pipe is reserved for error messages. It
goes by the name standard error. Windows picked up this standard and we can
access the output produced by either standard input (stdin) or
standard error (stderr). In the above example we caused the system to
capture the output from stdin and stderr so we can read them later.
The communicate() method returns a tuple with stdin and stderr
output, but it captures the output as a stream of bytes, not as a string. That
makes using the output difficult.
The last part of this script converts the bytes into a proper string. The line used is called a “comprehension” (just to make your head hurt!) What it does is to scan the list of bytes, and stick each one onto the end of a string. When the smoke clears, we have the string we wanted.
What this entire script did was to start up a separate program from
within our running script, let that program do whatever it normally does, and
capture whatever output was produced. We got a tuple back, containing the output
from the stdin pipe, and the stderr pipe!
I know this will be new to most of you, but it is a valuable thing to know. With it, we can build scripts that run programs for us, capture and process the output, then do other things as well. This can be a powerful way to build complicated scripts! I use this technique to build parts of my course notes. Embedded in my notes are notations that I capture and use to run programs. I collect the output from those programs and put that output back into the notes for you to see. Neat!
File redirection¶
Related to this scheme is your operating system’s ability to use
redirection when you run programs on the command line. You can use
these symbols:
So we could have used this to capture the Python version number:
python --version > version
You will now have a file on your system named version. That file has the
string we saw earlier in it!
This technique is often used to automate simple programs and capture the output. We will see more use of these techniques in later scripts.
Running your program¶
Suppose your program was not to be found on the system PATH, could we still
launch it using this technique?
Sure, just form the full path to the executable file you want to launch.
Suppose I need to launch the version of python that is living in /usr/local/bin on my Mac:
import subprocess
import os
pypath = "/usr/local/bin"
pyprog = os.path.join(pypath,"python")
p = subprocess.Popen(
[pyprog, "--version"],
stderr=subprocess.PIPE,
stdout=subprocess.PIPE)
output = p.communicate()
res = output[0]
ver = "".join( chr(x) for x in res)
print("out:",ver)
$ python ex6.py
Traceback (most recent call last):
File "ex6.py", line 7, in <module>
p = subprocess.Popen(
File "/usr/local/Cellar/python@3.8/3.8.6/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 854, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "/usr/local/Cellar/python@3.8/3.8.6/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 1702, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/bin/python'
It looks like the default version of Python on my Mac is the older version.
Useful Scripts¶
How you use your system, and what you need to do on that system is a very personal hing. I teach programming, so a lot of the tools I create are designed to help programmers get their work done.
Finding Program Source Code Files¶
In my C++ classes we build programs with several source code files. These files are not usually sitting in one directory. They may be buried in subdirectories set up to keep all the code about one aspect of a big project organized.
As an example of a script I wrote to help find app the C++ files under some folder, here is one I use all the time:
import os
import sys
dir = sys.argv[1]
ext = sys.argv[2]
result = []
for root, dirs, files in os.walk(dir):
for name in files:
if name.endswith(ext):
result.append(os.path.join(root,name))
print(" ".join(result));
This script takes two parameters. The first is the path to the top directory I want to search, and the second is the extension string I want to search for. The script prints a list of all file names found, complete with the path to each file. This list is separated by spaces, since I use the list as part of a building system you will learn about in a later course.
Here is what it looks like running:
Generating Images for Lectures¶
Here is a test script I use to generate image files containing nice figures for use in my lecture notes. The tool I use to produce these images is called TikZ, with depends on the LaTeX program to generate a PDf file Another program converst the PDF file into an image file I can use in my lectures.
To build my image file, I need to write a short piece of code in the TikZ language. To keep that code easy to write, I strip away all of the surrounding text needed t build a PDF file, leaving something that looks like this:
\draw[style=dashed] (2,.5) circle (0.5);
\draw[fill=green!50] (1,1)
ellipse (.5 and 1);
\draw[fill=blue] (0,0) rectangle (1,1);
\draw[style=thick]
(3,.5) -- +(30:1) arc(30:80:1) -- cycle;
To get this file ready for LaTeX processing, I need to surround my code text in a standard pile of text that never changes. So, I set up this file as well:
\documentclass[preview]{standalone}
\usepackage{tikz}
\begin{document}
\begin{tikzpicture}
%s
\end{tikzpicture}
\end{document}
What I need now is a simple bit of Python to put these two files together:
base = open('base.tex').read()
fig = open('sample.fig').read()
out = base % fig
fout = open('figure.tex', 'w')
fout.write(out)
fout.close()
Here is the combined file, ready for processing by LaTeX:
\documentclass[preview]{standalone}
\usepackage{tikz}
\begin{document}
\begin{tikzpicture}
\draw[style=dashed] (2,.5) circle (0.5);
\draw[fill=green!50] (1,1)
ellipse (.5 and 1);
\draw[fill=blue] (0,0) rectangle (1,1);
\draw[style=thick]
(3,.5) -- +(30:1) arc(30:80:1) -- cycle;
\end{tikzpicture}
\end{document}
The output from LaTeX is messy, so I am not showing it here.
Note
LaTeX is a powerful tool freely avialable on all systems, and one oyu may need to larn about if you go to graduate school. Many schools want a Thesis or Dissertation to be processed by LaTeX so the result will end up a nice book!
To test the code needed to build the image. I use this scrtipt:
import os
status = os.system('python fig_generator.py')
assert status == 0
status = os.system('pdflatex --interaction=nonstop figure')
assert status == 0
status = os.system('convert -density 300 figure.pdf -quality 90 figure.png')
assert status == 0
I do not need to capture the output from these commands. So, I can use another tool
in the Python os library to just run the program.
Those assert lines are a nice way tomake syre the program actually worked
before trying the next step. If the assert line failes to produce a
True`, the program will halt with an error message.
And, finally, here is the image produced:
That is a lot nicer than fighting with some graphics tool and trying to build something like this!
Python can be your best friend as a developer.
