Today’s @steemitblog post is brought to you by Steemit’s Blockchain team.
We have been extremely busy over the past several months, and wanted to share with you some of the things we have been up to. This post includes news of our plans for an improved bandwidth formula, updates on AppBase, RocksDB, and Hardfork 20 (HF20), as well as the latest developments for Smart Media Tokens (SMTs).

Scalability

As many of you already know from our previous scalability post, the Blockchain team has been very focused on scalability over the past year. We know that these types of changes are not as exciting as new features and platform enhancements, but ensuring that the blockchain is ready to scale to 100x or even 1,000x usage is something that is important to do before we actually scale to that degree. Neglecting scalability until it is actually needed is a recipe for disaster.

AppBase

AppBase provides a robust foundation for meeting all of our future scaling needs, and will allow us to grow the platform while at the same time managing the resource requirements for third-party application developers, witnesses, and exchanges to grow along with it.

It does this by enabling many components of the Steem blockchain to become modular by creating additional non-consensus blockchains as dedicated plugins. These plugins can be updated much more rapidly because they do not require replaying the entire blockchain.

The pre-release for AppBase was announced about three months ago and we appreciate all of the testing that the community has done since that announcement. We have also been testing extensively, and have been working on several changes to resolve some of the minor issues that were detected/reported. We are very close to having the official 19.4 AppBase release ready for witnesses and node operators to start safely using in production, and we will post as soon as it is ready.

RocksDB

As we mentioned in our Exploring Steem Scalability post, we have been spending a lot of time researching various ways to store the steemd data more efficiently. One of those approaches is using a technology called RocksDB. We are pleased to announce that we have decided to go with the RocksDB solution, and have already successfully converted the “account history” plugin to use RocksDB.

RocksDB is a fast-on-disk data store with an advanced caching layer, which could further minimize latency when reading/writing to and from the disk as it is optimized for fast, low-latency storage. Used in production systems at multiple web-scale entreprises (Facebook, Yahoo, LinkedIn), RocksDB is based on LevelDB but with increased performance thanks to its ability to exploit multiple CPU cores and SSD storage for input/output bound workloads. Its use in MyRocks, for example, led to less SSD storage use, longer SSD endurance, and more available IO capacity for handling queries.

In comparison with the previous account history implementation:

• An account history node can now efficiently be run by storing the state file on a nVME SSD drive, instead of having to keep the entire state file in RAM. This has allowed us to start running account history nodes on 32-64 GB RAM servers, instead of the 488 GB instances we were required to use before.
• An account history node can now re-index in about 10 hours, compared to the multiple days that it took before.
• The state file is much smaller, as RocksDB has built-in compression.

We completed extensive testing of these changes in the development/staging environments, and have been running the changes in production for a little over three weeks. Any users who have been querying account history data from the api.steemit.com endpoint over the past three weeks have been getting their data from the RocksDB plugin.

Account history was used as a test of the RocksDB technology to determine if it fits our needs. We are more than happy with the results of this test and are working on a drop-in replacement for Chainbase that relies on RocksDB instead of memory mapped files. This will dramatically improve the performance of steemd, and we are very excited to complete the transition.

Bandwidth

As we continue to scale the blockchain to more and more users, the bandwidth formula that we use to allocate resource usage across all of those users becomes more and more important. The bandwidth formula that we are currently using has been adequate for the level of usage we have had so far, but there is a lot of room for improvement. Our goal is to try to find the right balance between allowing new users to have an amazing experience using Steem-powered applications like steemit.com, while at the same time preventing them from using an unreasonable amount of the network’s resources or spamming the network. We would also like to simplify the mental model behind understanding how much Steem Power is required for certain levels of use.

Our current bandwidth formula makes a somewhat crude approximation of the cost of a transaction, based on the size of the transaction. While size is one important metric, the improved bandwidth formula should try to take into consideration all of the different resource constraints that a transaction may place on the network.

We have been researching ways to classify transactions based on their impact on several different factors. By taking all of these items into consideration, we hope to come up with a much better representation of a transaction’s true cost with respect to:

• Blockchain history size
• Reindex time
• State file size
• Memory usage
• Disk iops
• Network bandwidth

We are currently working to integrate a new tool called StatsD that measures statistics into steemd, so that we can acquire superior metrics. We are also researching different bandwidth implementations that can be used to allocate usage based on these metrics.

Once we have gathered all of the necessary data and settled on the best bandwidth algorithm design, we will share all of the details with the community.

Miscellaneous changes

Security Changes

The team has worked on several security patches to improve the stability of the network, which were released under Steem 0.19.3. The majority of witnesses and node operators have already picked up these changes and are running them in production. If any node operators are still running version 19.2, it is recommended that they upgrade to 19.3.

cli_wallet testing

The cli_wallet is a tool that is used by many witnesses, exchanges, and application developers to interface with the Steem blockchain. As we continue to make changes to the blockchain architecture (such as with AppBase and RocksDB), we felt it was important to design a suite of tests that could be used to ensure that the new version of code remains backwards-compatible with the old version, and doesn’t break any functionality with the cli_wallet.

It should be noted that once SMTs are released, the cli_wallet tool will remain backwards-compatible with previously existing functionality, but it will not be upgraded to support the new SMT functionality. We will be providing a new tool before then (and instructions on how to use it) that will act as a full replacement for the cli_wallet, as well as support all of the new SMT functionality via the command line.

Code style guidelines

In GitHub issue 2366 we are working on a code style document, so that developers have a guideline to inform style decisions and the codebase has a unified set of style rules. These will be useful for any developer making contributions to steemd, and will help keep the source code clean and reliable.

Transaction confirmation API

On rare occasions an action made on Steemit.com appears to work, only to disappear a few seconds later. There are some edge cases where the current transaction submission logic breaks down and results in this undesirable situation due to the transaction not making it into an accepted block.

We are working on a new API to better determine the status of a transaction. It is not only more efficient than the current transaction submission process, but will allow steemit.com to detect these scenarios and ensure your actions make it on the blockchain.

The discussion for this has been ongoing here, and our goal is to have a design finalized soon, so we can start work on the implementation.

Hardfork 20

Hardfork 20 has been on the back burner for a while as we focused on scalability-related solutions, but it is time to put it back on the front burner. We don’t have an exact date for the hardfork yet, but we are targeting early Q3 of 2018. More details will be shared on HF20 as the development progresses.

SMTs

We have several full-time developers dedicated to working on SMTs, and a lot of progress is being made. While many of the changes being worked on so far are highly technical, and basically serve the purpose of updating much of the existing functionality in steemd (which was designed for a single token: STEEM) to work for multiple tokens, there is still a lot of interesting functionality that is beginning to take shape.

Here are 10 of the interesting changes that have been completed so far:

• In 1508 the initial work was done to allow the creation of a new SMT.
• In 1653 and 1729, test cases were created to ensure that all of the SMT creation logic works as expected.
• In 1683 the foundation was laid for SMTs to integrate with the internal market, so that users can trade SMTs for STEEM and STEEM for SMTs.
• In 2029 the structure that will be used for the SMT Market Makers to run on the internal market was defined.
• In 1682, the transfer operation that is used to send tokens between accounts was updated to support SMTs.
• In 1843 posts/comments were updated to allow users to specify up to two SMT tokens for which the post/comment will be eligible (in addition to STEEM).
• In 1856 the vote operator was updated to support vote operations on posts/comments that include multiple voting assets (SMTs).
• In 1896 the operation for users to claim tokens after a post/comment that includes SMT payouts is paid out was created.
• In 2056 support was added for the smt_refund_operation, which can be used by contributors of an ICO to (optionally) cancel their contribution to an ICO and receive a refund if the ICO launch date is postponed.
• In 2021, 2160, and 2085, support was added for “vesting” SMTs (SMT Power).

More Technical Details

For those of you who are interested in the technical details, the team is also spending a lot of time on important design decisions in addition to coding. Many of these discussions are documented in GitHub.

One example can be found in this issue, where we discussed how to handle automatic actions such as SMT emissions and Market Maker transactions. In another, 2212, we discussed the corner cases of SMT vesting to prevent integer overflows and rounding errors.

Testnet

We know that everyone is eager and excited to have SMTs completed and launched into production as soon as possible. So are we! Our dedicated team is working around the clock to make this happen. The first major milestone we are aiming to achieve is to have an SMT testnet (called “Forerunner”) up and running, where developers can start to play around with some of the implemented features.

We will continue to keep you up to date on our progress and will let you know as soon as the Forerunner testnet is ready for use.

Steem on,

The Steemit Blockchain Team