This past week has been a relatively rough week for witnesses across the board, but running Steem Engine and Splinterlands has made it especially grueling. These are businesses and we make money on the chain. In the case of Matt and I our livelihoods are completely dependent on a functioning Steem blockchain. When the chain goes down we have no products or services to sell and we also get slammed with concerned customers while working to restore the chain. It's a high stress time and an exhausting time for any period, but especially when it extends over the better part of a week.
tl:dr - seed.steemmonsters.com
Seed.steemmonsters.com is a seed node with blocks, account info, and broadcast plugins enabled which allows it to get a handful of key functions of an RPC up and running in short order. It's sitting behind jussi and nginx so it can handle most calls sent it's way. If done with SIAB enhancements from Privex it can be setup like this out of the box with easy to follow prompts and commands.
It's at least ~24 hrs for a full Mira replay, but runs on a $50/16gb server. The non-mira version replays in 4-6 hours on a $100/64gb server.
Apps that don't need post/vote data or account history can run completely off of a $50 or $100/month server. Though there are still lingering issues in the ecosystem to figure out if even that's enough.
Splinterlands during the hardbork
Splinterlands went from the all time high number of games played in the new rules (which were introduced in Feb) to the second lowest number of games played in company history.
We also went from the most number of players in company history to the fewest number of players in company history.
A week of difficulties
The challenges weren't just a single day of chain halt, but persistent for roughly a week with down time, limp time, and up time. This causes major challenges with game play, getting players to be there consistently, sales slow down or don't go through, and even just getting things to work at all is challenging for the players and devs.
Obviously when the chain is halted nothing on Steem happens. However, during partial uptime some things are possible, but we learned the hard way that as things currently stand the ecosystem is still extremely reliant on Steemit RPC nodes.
Some nodes were up but Splinterlands still wasn't really playable
The back-end for the Splinterlands game has been primarily running off of the RPC node set up by @anyx for some time now, and it has a robust system to automatically switch between various other available RPC nodes if one should experience issues.
Roughly 25% of all transactions are now coming from Splinterlands and Steem Engine so we've been working to reduce our footprint on steemit nodes (25% by raw number of transactions, but our transaction are also very low memory/rc usage so they relatively inexpensive to the chain).
Based on features of the anyx.io RPC, we thought we were covered in the event that the Steemit RPC nodes went down, but unfortunately that wasn't the case. While our server was still running and processing blocks (using the Minnow Support RPC which was one of the first ones back up), most players were unable to broadcast any transactions since tools like Steem Connect and Steem Keychain all use api.steemit.com by default :-(
@stoodkev is currently hard at work updating Steem Keychain to support things like automatically switching to a different RPC node if the current one is not working, and allowing apps to specify an RPC node to use with each operation. We hope to see Steem Connect and other tools do the same so that in the future a Steemit RPC node downtime will have little to no impact on the ecosystem.
Getting Nodes
Even with updated tooling, there are still a pretty small number of usable RPC nodes available in the ecosystem, which is not ideal. One main reason for this is that the focus is often on providing full nodes, which include all or most of the available plugins. This either makes the nodes very expensive to run, tricky to run, and have unreasonably long replay times. The last part is also a business killer when the chain needs replays.
Many apps and tools don't actually require a full node if they aren't using vote/post operations or need account history. I spent some time investigating what is needed at a minimum and what resources are required to run apps like Splinterlands and Steem Engine that don't need those plugins...
Mira
Mira is an amazing advancement for the chain, but it also comes with a major difficulty: replay time. If you happen to have a saved state file it can shorten the time significantly, but not everyone saves them and not every save point will still be valid for replays. The benefit of course is that server costs can be drastically reduced because the reliance on expensive RAM is decreased.
Non-Mira
While Mira is great for high mem RPCs it doens't do much for cost savings for withness/seed server operators. The main advantage of non-mira boxes is that they replay quickly. Instead of a 24hr+ replay time for a witness or seed these can replay in 4-6 hours.
Needs
What we ultimately decided that we needed was a node that could broadcast, get blocks, and get account info. We don't actually need a full RPC and it turns out we don't even need a high memory or hivemind version nor do we need account history. Those features take lots of RAM and/or take a long time when replaying. It turns out what we need is pretty simple.
Privex
After scoping lots of options I defaulted to the best answer out there: Privex. Not only does privex have reasonably priced servers but over the years @someguy123 et al have been extremely helpful team members to me personally and continue to develop tools to make running Steem easier. He's a fantastic asset to our business, this blockchain, and the service provided is above and beyond "cheaper" virtual servers out there.
He already had a Seed node operational with a few things running on it- https://steemseed-fin.privex.io/. Testing showed that it wasn't quite up to speed for our purposes. It needed Jussi and nginx to start working nicely.
Mira-Witness Node Converted
As it turns out I have 3 witness nodes. 2 non-mira nodes and I've rotated in the backup mira-node a few times. We decided to cannibalize the node. It's a seed node with a few extra features (or an RPC node missing a lot of features). Someguy took a moment and added some pretty handy features to make jussi and nginx work out of the box.
The node now has get account info, blocks, and broadcast enabled. It's behind both Jussi and nginx. This should serve as a Compact Broadcast Node.
Works currently with Steem Monsters and Steem Engine; most of the functionality of Keychain
So far I've been able to test this box on Steem Monsters. I purchased a pack using keychain and then opened it.
I've managed to make a few steem-engine market purchases with it.
So far it does everything I normally do on these apps except it can't show me Steem history. It says "errors." That's a very small price to pay for an extremely light node that with some changes in the rest of the ecosystem could get us up and running very quickly after a hardbork.
Major Break Through
My grandmother always used to say "When working with blockchain technology and life gives you lemons it's time to make a low cost, functional, RPC node that can service your current needs and be expanded to future chain apps." She was ahead of her time...
Steem continues to amaze me. Trying to do this on other blockchains could literally cost a fortune every month. Here, we can have our own compact broadcast node that can offer nearly all of the functionality we use in the game and for the exchange. It can help service the entire ecosystem. 24+ hr replay version costs $50/month and 4-6 hour replay versions cost $100/month.
I feel like this is a pretty good discovery that many apps that aren't focused on posting/curating, like games, can function on inexpensive broadcast hardware. Hopefully, Steem Monsters and Steem Engine can demonstrate this in the wild, and we'll help show others how they can eaisily and inexpensively build their apps on top of Steem.
Gratitude
This post is only possible after a lot of discussions with people from around the chain. Thank you to Justin from Steemit. I had a lengthy discussion with him trying to scope out various versions of this along the way. Thank you to my fellow witnesses especially smooth, blocktrades, gandalf, anyx, markymark, netuoso, drakos, and someguy123 who had various pieces of advice along the way. Narrowing in on our app needs and what I'm deeming high priority ecosystem wide improvements wasn't easy to weed through.
Thank you for your input! I'm hoping that the outcome is something that can really showcase how inexpensive and easy it can be to build apps on Steem.
What's next?
Now that we have a compact broadcast node up and another on the way soon the next piece is to campaign to fix the rest of the ecosystem. The libraries and a bunch of tooling need upgrades. We want to see the blockchain functioning even though steemit nodes are not. If everything remains partially or completely dysfunctional because the rest of the ecosystem can't function without steemit nodes then this blockchain is entirely too centralized and dependent on Steemit. Updating libs and tooling to allow multiple RPCs and seeds to broadcast and also support failover doesn't appear to be an extensive amount of work for any one project, but I think the challenge in this particular case is that it needs to happen in lots of places over the whole ecosystem.
I'm working on it and focusing on it. Should be fun!