In my May witness report I gave you a teaser for this technical article. I will explain how I use a combination of tools to keep tabs on my witness nodes and give you the practical benefit of what I've learned in putting this into practice.
This explanation is not intended to be an exhaustive tutorial on each of these tools, however it will cover the elements not so easy to find documentation for and that I found difficult and time consuming to figure out. I will also provide a link to my github repo where you can find the scripts I used.
Summary of Features
1 - Automates startup of witness node upon system restart
2 - Automates starting cli wallet when witness becomes synchronized with blockchain
3 - Optionally unlocks cli wallet to support bts_tools price feeds
4 - Creates a multi-windowed tmux session to view witness log file in real time
5 - Starts bts_tools monitoring and associated web interface with price feeds
6 - Creates a shell window for running builds or for general use
7 - Attaches multi-window tmux session upon logging in to server via ssh
8 - Provides administration of services through systemd (start, stop, status, journalctl etc)
An image of the tmux session is shown above. Below is a screenshot of the web interface provided by @wackou's bts_tools which I have been using since I first began producing blocks in 2015. His tools are built with python 3, use a flask template system for the UI and served to the web via nginx through a uwsgi interface. Web access uses ssl encryption and password authentication to provide "at a glance" monitoring from anywhere the web can be accessed. I couldn't recommend @wackou's bts_tools more highly.
We had some grand plans to work together and establish an infrastructure of seed and witness nodes that has never come to fruition. Back in the day I wrote a python extension to his bts_tools to distribute delegate (i.e. witness) pay to a set of accounts specified in the config file. I used it for the duration of the BTSX but it was never updated to work with graphene. Since graphene was released there has been little need for it. It wouldn't take much to get it working with graphene should anyone need that functionality.
Components Used
Subsystems
o - Graphene witness_node -- blockchain software
o - cli_wallet -- Command LIne interface for witness_node
o - bts_tools -- Nicholas Wack's marvelous tool suite
o - nginx -- web server to support bts_tools (and anything else you might want to serve up)
Scripts
o - witnessSysD -- installs a SYSTEM scoped systemd unit to manage witness node service
o - tmuxSysD -- installs a USER scoped systemd unit to manage tmux service
o - startTmux -- initializes tmux session, starts bts_tools and cli wallet, monitors witness RPC socket
There are a few other items in the github repo you may find useful but these are the main elements involved. Note there are a few lessor items I use that will make your life much easier such as virtualenv and virtualenvwrapper that provide a "sandbox" for python scripts. There are also a number of almost forgotten bumps in the road you may encounter such as how to get different python versions to coexist in harmony. It's also worth noting that some items such as uwsgi can be installed globally OR via python pip in a virtualenv sandbox. I recommend the later as it will isolate and encapsulate the dependencies of bts_tools.
SystemD
As I mentioned before, I had great difficulty getting systemd working for a USER scoped session. It was much easier using systemd for SYSTEM scoped service units. A unit in systemd parlance is a unit of work or a service to run. It can be as simple as a script or it can get very involved as to dependencies and order of execution. I started using only SYSTEM scoped units and later migrated the tmux elements to a separate USER scoped service unit as that was oriented towards user interaction. I also had difficulty with tmux in running the witness node and user items which them separated eliminated (I never got to the bottom of why tmux thought I was trying to run nested [tmux] sessions. I believe it should be possible to run everything in a systemd USER session, but that is not where it is today). All USER scoped systemd unit files reside in the user's home directory under ~/.config/systemd/user.
The primary obstacle I had to overcome to allow systemd USER sessions was to get a USER scoped dbus process fully started. Although systemd did start a user session and create a /run/user/<uid> folder, it was totally empty. This is where the dbus sockets live for the USER session (actually /run/user/<uid>/systemd/notify & private). Usually this is accomplished by the operating system via the X Windows startup process. However when that is absent some other means is required, for example when logging in though an ssh connection. When the user session is properly initialized two environment variables are set:
- XDG_RUNTIME_DIR=/run/user/<uid>
- DBUS_SESSION_BUS_ADDRESS=unix:path=$XDG_RUNTIME_DIR/systemd
I had always configured my sshd_config script to disallow password authentication as well as disabling PAM with UsePAM = NO. I still disallow password authentication, however PAM (Password Authentication Module) must be enabled so that libpam-systemd will be used to initialize the user's systemd context, which includes creating a separate dbus user process, in memory message sockets and setting environment variables so applications like systemctl --user can communicate. Another element I had to find was that on some platforms (I primarily use Debian and Ubuntu operating systems) the dbus sockets were still not being created. I noticed that on platforms that did work properly there was no /etc/pam.d/atd file present. I'm not sure of what that file does, but removing it resolved the issue and the systemd user session was properly established. I suspect atd is used to delay the start of "at" (i.e. cron style) jobs, but I am not certain of that.
Another aspect hindering the continuous operation of USER scoped units after logging out is a systemd setting, whose default behavior changed with version 206. The new default behavior is ALL user processes are terminated on logout. It was a highly controversial change and you can find many discussions about it online. There are several solutions, the one I use is a script command loginctl enable-linger <user> which reverts the default for the user account specified.
To automatically switch to the tmux multi-window view upon logging in you will need to add the line tmux attach-session bts-node to your .profile script, or whatever your shell of choice runs when it starts.
I think that covers the main points, and along with the scripts I published in github and with bts_tools provided by @wackou you have the essential elements to reproduce these interfaces.
verbaltech2 and baxters-sports-witness
are always appreciated! Thanks for your time and attention
