bts_tools release 0.4

Version 0.4 of the bts_tools has been released, with full support for Steem!

Here's the changelog:

0.4 (2016-05-01)

• API CHANGE: complete rework of the configuration system, please delete
  your old config.yaml file if you have any
• added full support for Steem, including feed publishing
• added feed publishing for BTS assets: TUSD, CASH.USD, CASH.BTC, ALTCAP
• added monitoring plugin that checks on the amount of free disk space
• fixed missed block notification for Graphene clients
• logs are now also present as rotating logfiles in the ~/.bts_tools folder

And for new people that have not known or used the tools before, here's a quick executive summary of the tools:

bts_tools tutorial

The bts_tools are meant to build, run and monitor full nodes (witness, seed node, ...) for graphene-based chains.

The first step you will have to do is install the tools:

$ pip3 install bts_tools

Building and running are quite easy:

$ steem build    # builds the latest master version
$ steem list     # lists all the binaries you have built
$ steem run      # runs the last compiled witness client
$ steem run_cli  # runs the last compiled cli wallet

You'll probably ask yourself: "how does it do all that? how can it know which witness I want to run? where does it get the private keys from?"

And you would be right, there's a piece missing, which will also bring us to monitoring, which is the config.yaml file. This file contains all the configuration you need for running the tools, and resides in ~/.bts_tools/config.yaml. The config.yaml file is actually quite short now (compared to previous versions of the bts_tools), as a lot of the (mostly static) values that were there before have been moved to a separate "default" config file, which can be seen here.

At this point, I would suggest to open ~/.bts_tools/config.yaml and have a look inside. If it doesn't exist yet, just run any of the previous commands, which will have the side-effect of creating a default one (steem list should do, for instance)

When running the bts_tools once, it will also write a ~/.bts_tools/full_config.yaml file containing your config.yaml values merged into the default ones. This is quite useful for debugging and/or having a look at all the options that you can play with.

After setting the correct values in the config.yaml file, running the monitoring webapp is a simple matter of:

$ steem monitor

et voila! go to http://localhost:5000/ and you should see the dashboard of the clients currently being monitored. The tools will also notify you via email or boxcar if anything goes wrong (block missed, client going out-of-sync or crashing, no more disk space, etc.)

There are a lot more details to know about, but this should get you started. More info can be found in the documentation (not updated yet for Steem as of the 0.4 release, hope to get it done by 0.4.1).

NOTE: One tip that should prove useful for some is if you want to build the Steem client as low memory node and without the dependency on Qt for patching posts, add this to your config.yaml file:

build_environments:
    steem:
        cmake_args: ['-DENABLE_CONTENT_PATCHING=OFF', '-DLOW_MEMORY_NODE=ON']

And even if you don't plan on running the monitoring tools, just the convenience of steem build and steem run should convince you of their usefulness.

A roadmap and potential ideas

The main objectives that I try to accomplish by developing the tools can be summarized like that:

• build, run and monitor graphene clients, and ensure they keep running smoothly
  => this is pretty much the current status of the tools

• provide tools/infrastructure for witnesses during critical situations (eg: the fork of Joseph)
  => TODO, these are my current thoughts, but I would like this to be a brainstorm between witnesses as how to best react in the case of forks

  • publish list of checkpoints to get back on main chain
  • publish dumps of chain folder
  • revive the chain server mechanism of BTS1, which allowed a node to sync the blockchain not through the p2p network, but by connection to a given host and downloading the blockchain directly from it. This had the nice side-effect to ensure that you would end up on the same fork as the chain server one, so as long as there was one node on a fork that also had the chain server functonality activated, it was easy to revive a forked network. Unfortunately, this has been dropped in the switch to BTS2.0, but as forking was much less common than before, it hasn't been really noticed.

• provide tools/infrastructure for witnesses during normal situations

  • deploy tools [90% done]
    I currently have a deploy script that I used to deploy the seed node here using the single command steem deploy_node deploy_config.yaml <ip_addr_of_fresh_vps>. The configuration is still a bit brittle, but I plan to polish it and release it shortly
  • DDoS protection: doing a network timing analysis, it would not be very difficult to find the IP addresses of the witnesses, which could lead to a DDoS on them. I made a proposal with Thom at the end of BTS1 to setup a mechanism to hide the IPs of the witnesses (think: an extremely simplified version of Tor, but without the added latency, we need to have 3 seconds block after all!), but I never got around to implement it (BTS2 had been released and all efforts went to porting the tools to graphene)

So this is pretty much my wish list of immediate needs, I do have more ideas and stuff in the pipeline, but let's get the basics done first, and we'll move to the crazy stuff later!

If this is all something that you would like to see, please don't forget to vote for witness wackou. Thank you!