SSH Remote Control

Note

Last Updated on 2024-03-27

Read time: 13 minutes (3286 words)

As part of my research work, I maintain several small clusters.

Cluster Systems

Processor	Num	Cores
Intel-I5	4	8
Pi 3	4	16
Pi 4	7	14
Pi 5	4	20

That is a lot of machines, fortunately most of them were fairly cheap and acquired over several years of teaching. My problem is managing the setup and code on all of those critters!

All of these systems run some form of Linux. I have all set up for SSH access with no password required. My intent is to use these clusters as test beds, to experiment with parallel processing techniques. Logging in to a console on each machine using SSH is simple enough, but repeatedly login in and running commands on a bunch of machines is painful. I do use the Ansible tool to manage basic configurations on all of these machines, which significantly improves my life.

Developing for a cluster

In most cases the code that runs on a cluster is written in some high level language like C++ or modern Fortran. I use a MacBook Air as my primary laptop, and it is all set up to produce documents (like this one) and build Python applications. My goal is to use Python to create a simple automation technique that lets me send a list of commands to one or more machines and collect the console output for display on my management laptop. That output will be included in my documentation using the Sphine programoutput extension.

The inspiration for this experiment is Ansible, but I want to have more control over the command sequence and the collection of output from a remote task.

Time to improve the process.

Test Setup

I have a simple Raspberry Pi 3 running on the top of my office bookcase. That machine has been running for over a year and can be accessed using a Wifi connection.

In this example, I have set up an alias for the machine (mine is picam) that dets up the remote machine IP address. I have also added my SSH public key to the known_hosts list on the Pi. This allows logging in with no password.

Once all of this has been set up, this should work:

$ ssh picam 'uptime'
 18:28:46 up 247 days,  4:43,  2 users,  load average: 0.91, 0.91, 0.91

SSH supports running commands from a text file. Here is an example command file:

$ cat ex2.txt
uname -a
df -H
uptime

The normal SSH command to run this locally is:

ssh picam < commands.txt

Unfortunately, the Sphinx extension I use to include program output in my documents cannot handle file redirection like this. We could transfer the command file to the Pi and run it there but we would miss the individual command output results. A solution (maybe not the best) is to create a simple local Python program that reads our command file and runs the SSH commands for us. (Python is great for creating simple automation tasks like this (see Weigart [Wei07]).

Here is a basic Python snippet using this approach:

import subprocess
import sys

subprocess.run(['ssh','picam', 'uptime'])

And here is the program output:

$ python code/ex3.py
 18:28:46 up 247 days,  4:43,  2 users,  load average: 0.91, 0.91, 0.91

This works! We now need to wrap this up in a loop that reads a line from the command file and creates the list of command parts. Python is up to this job!

uname -a
uptime
df -H

$ python code/ex4.py code/ex1.txt
Darwin MBair-M1 23.4.0 Darwin Kernel Version 23.4.0: Fri Mar 15 00:12:41 PDT 2024; root:xnu-10063.101.17~1/RELEASE_ARM64_T8103 arm64
18:28  up  2:57, 2 users, load averages: 1.45 1.49 1.46
Filesystem        Size    Used   Avail Capacity iused ifree %iused  Mounted on
/dev/disk3s1s1    494G     10G    382G     3%    404k  3.7G    0%   /
devfs             203k    203k      0B   100%     686     0  100%   /dev
/dev/disk3s6      494G   2148M    382G     1%       2  3.7G    0%   /System/Volumes/VM
/dev/disk3s2      494G   6210M    382G     2%    1.4k  3.7G    0%   /System/Volumes/Preboot
/dev/disk3s4      494G     20M    382G     1%      72  3.7G    0%   /System/Volumes/Update
/dev/disk1s2      524M   6312k    506M     2%       3  4.9M    0%   /System/Volumes/xarts
/dev/disk1s1      524M   6459k    506M     2%      31  4.9M    0%   /System/Volumes/iSCPreboot
/dev/disk1s3      524M    549k    506M     1%      39  4.9M    0%   /System/Volumes/Hardware
/dev/disk3s5      494G     92G    382G    20%    1.4M  3.7G    0%   /System/Volumes/Data
map auto_home       0B      0B      0B   100%       0     0     -   /System/Volumes/Data/home
******************************
******************************
******************************

Generating this code was not too difficult, considering the need to read the command line, make sure the lines were not blank, then creating a list of final command parts to be proceeded by the subprocess.run code.

However, the output displayed here does not currently match what I see when running this code locally. It appears that output buffering by the Sphinx extension is not dealing with this well. A solution is to generate the output lines in the Python code and dump that output at the end of the code.

uname -a
uptime
df -H