I have been building a blockchain parser for my Steem Fundamental Analysis project, and I thought it would be a good idea to document my findings.
If you're a developer and you'd like to build an app on top of Steem/Steemit, this guide is for you.
Setting up your node
First, we need to setup a node. There are plenty of guides and tools (for Docker, Ubuntu, Windows) on steemit on how to do that, so I won't repeat myself here.
To get access to all the API's, modify your config.ini to have the following values:
enable-plugin = witness account_history tags follow market_history
public-api = database_api login_api network_broadcast_api follow_api market_history_api
Also, you should remove miner and witness fields. We aren't interested in mining, just running a node.
Now you can run steemd:
./steemd --rpc-endpoint
The --rpc-endpoint will setup a websocket endpoint at ws://127.0.0.1:8090.
You should be able to connect to it from your favorite programming language (Python, JavaScript, etc.), or by using a tool such as wssh.
If you're using Python, then I suggest the awesome python-steemlib by @xeroc.
You can connect to the RPC like so:
from steemapi.steemnoderpc import SteemNodeRPC
rpc = SteemNodeRPC("ws://127.0.0.1:8090", "", "")
Analyzing the Blockchain
Now that we have a node running, and we are connected to it, we can issue commands.
In this tutorial, we will be using the following RPC commands:
get_block()
get_dynamic_global_properties()
Again, the RPC is available for us at ws://localhost:9080.
The full list of commands is available here (scroll to the bottom):
https://github.com/steemit/steem/blob/master/libraries/app/include/steemit/app/database_api.hpp
In order to get a block from the blockchain, we want to call a RPC command called get_block, which takes just one argument: Block ID.
Block ID is just a number, anywhere between 0 and the current block ID.
Each new block has an ID at an increment of 1.
You can find out what the current block ID is by looking at steemd output:
651068ms th_a application.cpp:439 handle_block ] Got 1 transactions from network on block 3695832
Or by asking steemd via RPC:
props = rpc.get_dynamic_global_properties()
last_confirmed_block = props['last_irreversible_block_num']
So to traverse the entire blockchain, from start to finish, we would do something like this:
props = rpc.get_dynamic_global_properties()
current_block = 0
last_confirmed_block = props['last_irreversible_block_num']
while current_block < last_confirmed_block:
current_block += 1
print("Processing block %d" % current_block)
block = rpc.get_block(current_block)
pprint(block)
Now lets look at what get_block gives us. If we call rpc.get_block(3332318), we get a block 3332318 in a json format, that looks like this:
{'extensions': [],
'previous': '0032d8dd7fef5c0eafd7667805134effb9ef21e9',
'timestamp': '2016-07-19T13:29:15',
'transaction_merkle_root': '5be8c3f95a29253153e911fd079528ad8216958a',
'transactions': [{'expiration': '2016-07-19T13:29:24',
'extensions': [],
'operations': [['vote',
{'author': 'beowulfoflegend',
'permlink': 'how-steemit-self-polices-content',
'voter': 'jills',
'weight': 10000}]],
'ref_block_num': 55512,
'ref_block_prefix': 2485993294,
'signatures': ['20196c89f5e2ca58da3377a30ee08782df59dbd4b72db40ec3348f3417ff653fa84cf684748853e8ea43363c747c74b4ff015626ef9e2b2e902e0ee98a80195336']},
{'expiration': '2016-07-19T13:30:10',
'extensions': [],
'operations': [['comment',
{'author': 'weenis',
'body': 'This is Amazing! Upvoted! \n',
'json_metadata': '',
'parent_author': 'begstreets',
'parent_permlink': 'abstractions-no-1',
'permlink': 'abstractions-no-1',
'title': ''}]],
'ref_block_num': 55516,
'ref_block_prefix': 765250389,
'signatures': ['1f7711cfc441bbc5e6df16d7b73ea7d4ea7d7a40a3c3a1fd857d67b50b30c139d06f742e5fd15a68f2109469c977a12c50abab81632b8a4ba5e328c5048290153b']},
],
'witness': 'riverhead',
'witness_signature': '1f4236f6a0ff79ab2ff9a190162aeb25b5e1509ca44df8803c8091a2abcfe2022f733eb77c15ec8265a1b1d0d2226038b3259db9ebe17b871bcbf3be0dee605bc9'}
We are mostly interested in transactions and operations. Transactions contains a list of operations, which are basically all the events that happened on the blockchain within the 3 second period included in the block.
Operation Types
Currently, the Steem blockchain contains the following operations:
- comment, delete_comment, vote
- account_create, account_update, request_account_recovery, recover_account
- limit_order_create, limit_order_cancel
- transfer, transfer_to_vesting, withdraw_vesting, convert
- pow, feed_publish, witness_update, account_witness_vote
- custom, custom_json
Blog Posts, Comments and Votes
This is a blog post:
=====> comment
{'author': 'rozu15',
'body': 'content omited becuse it breaks my post',
'json_metadata': '{"tags":["trump"],"links":["https://twitter.com/realDonaldTrump/status/755254384062263296"]}',
'parent_author': '',
'parent_permlink': 'trump',
'permlink': 'us-election-melania-trump-plagiarised-michelle-obama',
'title': "US election: Melania Trump 'plagiarised' Michelle Obama"}
This is a comment. It has empty title:
=====> comment
{'author': 'densmirnov',
'body': 'nice nickname',
'json_metadata': '{"tags":["steem"]}',
'parent_author': 'steemiscrap',
'parent_permlink': 're-cryptorune-steem-dollar-poloniex-pump-is-a-arbitrage-goldmine-20160719t133438897z',
'permlink': 're-steemiscrap-re-cryptorune-steem-dollar-poloniex-pump-is-a-arbitrage-goldmine-20160719t133602671z',
'title': ''}
The comment has been deleted:
=====> delete_comment
{'author': 'densmirnov',
'permlink': 're-cryptorune-re-densmirnov-re-cryptorune-steem-dollar-poloniex-pump-is-a-arbitrage-goldmine-20160719t133207807z'}
The comment has received a vote:
=====> vote
{'author': 'kain-jc',
'permlink': 'a-meditation-on-love',
'voter': 'cantinhodatete',
'weight': 10000}
If we divide the weight by 100, we get the % of the vote. For instance, a full upvote is 100%. The full downvote is -100%. Half the upvote is 50%.
Account Features
Account has been created:
=====> account_create
{'active': {'account_auths': [],
'key_auths': [['STM7huHJRrqHE6FDNHJEsUYNNKuQKBfr4iS3xwgyGzXDurbiCRx2V',
1]],
'weight_threshold': 1},
'creator': 'steem',
'fee': '5.000 STEEM',
'json_metadata': '',
'memo_key': 'STM7R3iScNyQCytvB8EcRdEYLfJCGp3UkyNm7Mt4c9BE1ZoUTdazw',
'new_account_name': 'saripdol',
'owner': {'account_auths': [],
'key_auths': [['STM8XzPFfu3yCujJiSbgdJvKm69LMcm1ckocuvoSaaWRmPRzfH7jr',
1]],
'weight_threshold': 1},
'posting': {'account_auths': [],
'key_auths': [['STM8KKqZufL2PM7V9dHeSW8KpN4wWs8cGaiQsz7FCrPohX1ZkfPvv',
1]],
'weight_threshold': 1}}
Account has been updated:
=====> account_update
{'account': 'signaltonoise',
'active': {'account_auths': [],
'key_auths': [['STM83ihutAefZSHCJRXLHiRnji8VhQHcouMJc6xGtaLW6PCARXiDa',
1]],
'weight_threshold': 1},
'json_metadata': '',
'memo_key': 'STM81hFsfm2csFyPYDCdwjuGJX33EB2Dx6rG74AfFgStu9C6PvzMd',
'owner': {'account_auths': [],
'key_auths': [['STM8fcgjrpKbVLHyy3RKnQGx3dhYEdY7bKhbv3FaML5ooK8PpdJaR',
1]],
'weight_threshold': 1},
'posting': {'account_auths': [],
'key_auths': [['STM75QK5e69SAuMzVtJqGex3HcaqrPcxpXgxb72WxbUJn2apRX7KX',
1]],
'weight_threshold': 1}}
Transfers, Vesting, Conversions
A transfer of SBD or STEEM from one account to another.
=====> transfer
{'amount': '34.428 SBD',
'from': 'thisischris225',
'memo': 'ee1d91d4036200c7',
'to': 'poloniex'}
Someone has powered-up:
=====> transfer_to_vesting
{'amount': '11.789 STEEM', 'from': 'leksimus', 'to': 'leksimus'}
Someone has powered-down:
=====> withdraw_vesting
{'account': 'giveandtake1', 'vesting_shares': '1184211.402324 VESTS'}
Someone placed a conversion order:
=====> convert
{'amount': '0.158 SBD', 'owner': 'jamessmith', 'requestid': 1468935121}
This order executes 7 days after its initiated, and takes the 7 day average price as the execution basis. It is useful for placing a big order in the internal market without moving it too much.
Mining, Witnesses
Finding a pow (mining):
=====> pow
{'block_id': '0032d98aaa917775f98859c3235f43286479e1b7',
'nonce': '6342739805603833088',
'props': {'account_creation_fee': '0.001 STEEM',
'maximum_block_size': 131072,
'sbd_interest_rate': 1000},
'work': {'input': '507906ad12f0d3bc57f636849dc117c5c970e79f0411e7e07459f78c09b350a0',
'signature': '209c919b436563ae30171356cb7fcb6677794c9cbf163c695d3ce56841a9c86f1d06ab7f4b28c4c493a5d951664b32b7fb76714b30e49c8035bf12c520b1c6c3df',
'work': '0000000350c2f2cae73dab00d1c2a1e2fafbf22e1e2fe0bb2a0a4e3a69c3b976',
'worker': 'STM574Rx1dML1K8Mi9Uz42yWDYAr6ymrxC723HR1YX81Jqk7MSaRT'},
'worker_account': 'redrockmining'}
Witness changes its settings:
=====> witness_update
{'block_signing_key': 'STM7yFmwPSKUP7FCV7Ut9Aev5cwfDzJZixcreS1U3ha36XG47ZpqT',
'fee': '0.000 STEEM',
'owner': 'anyx',
'props': {'account_creation_fee': '2.600 STEEM',
'maximum_block_size': 131072,
'sbd_interest_rate': 1000},
'url': 'https://steemit.com/witness-category/@anyx/hello-from-anyx'}
Witness publishes new market rates:
=====> feed_publish
{'exchange_rate': {'base': '4.100 SBD', 'quote': '1.000 STEEM'},
'publisher': 'steemed'}
Trading
Place internal market order:
=====> limit_order_create
{'amount_to_sell': '94.000 STEEM',
'expiration': '2016-07-19T16:10:40',
'fill_or_kill': False,
'min_to_receive': '260.000 SBD',
'orderid': 94,
'owner': 'ledzeppelin'}
Cancel internal market order:
=====> limit_order_cancel
{'orderid': 1468934907, 'owner': 'fabio'}
Account Sundry
Voting for a witness:
=====> account_witness_vote
{'account': 'leontyashka', 'approve': True, 'witness': 'arhag'}
Initiating account recovery:
=====> request_account_recovery
{'account_to_recover': 'stan',
'extensions': [],
'new_owner_authority': {'account_auths': [],
'key_auths': [['STM53xgdjGJ26QEF8qUbGsxguRBQU1UMqLKTT3LJ2ysgsLdr3zJk2',
1]],
'weight_threshold': 1},
'recovery_account': 'steem'}
Account recovery complete:
=====> recover_account
{'account_to_recover': 'val',
'extensions': [],
'new_owner_authority': {'account_auths': [],
'key_auths': [['STM7igHgvdVmUJFJadc56d3wuk28xPo9NV4qQLhsHBzmpiQvTq2Pe',
1]],
'weight_threshold': 1},
'recent_owner_authority': {'account_auths': [],
'key_auths': [['STM4wo7NZSUY86RDmaZrQuhuAybyCNSEfnpN2fHmKX7V4mQtPrsuK',
1]],
'weight_threshold': 1}}
Custom
Custom json currently only supports the follow plugin. More features to come in the future.
=====> custom_json
{'id': 'follow',
'json': '{"follower":"manugbr93","following":"natenvos","what":["blog"]}',
'required_auths': [],
'required_posting_auths': ['manugbr93']}
what can be either "blog","posts", "blog", "posts" or "".
I suspect "" means unfollow. I do not know what is the difference between blog and posts however.
An finally, this mysterious custom operation:
=====> custom
{'data': '046162696408637563697374696d03c6278df998dd7cf0bac77dc3af89debf1f628eebbe9e86daa9762b7590630218023560f4ac629975cda967a694a07ebea5d912fcc31bf732aca9908ec9c04d603850ae00e0fd3705003f8ea76c2073f7cac9ce38f97dd8d32ba5df47adbea29c29758d776467ce16f055a5095563',
'id': 777,
'required_auths': ['abid']}
@jl777 thinks its an encrypted private message.
Example: Parsing the Blockchain
Suppose we want to find out how many followers we have, and get the list of people whom are following us.
Here is how we can parse each block, only looking for custom_json operations to find out:
from steemapi.steemnoderpc import SteemNodeRPC
rpc = SteemNodeRPC("ws://127.0.0.1:8090", "", "")
MY_USERNAME = "furion"
props = rpc.get_dynamic_global_properties()
current_block = 0
last_confirmed_block = props['last_irreversible_block_num']
while current_block < last_confirmed_block:
current_block += 1
block = rpc.get_block(current_block)
if "transactions" in block:
for tx in block["transactions"]:
for opObj in tx["operations"]:
#: Each operation is an array of the form
#: [type, {data}]
op_type = opObj[0]
op = opObj[1]
if op_type == "custom_json" and op['id'] == "follow":
j = json.loads(op['json'])
if j['following'] == MY_USERNAME:
follower_count += 1
print("%s is following my %s, I have %d followers now." % (
j["follower"], j['what'], follower_count
))
I only have 8 followers, which makes me sad:
kilrathi is following my ['blog'], I have 1 followers now.
cloveandcinnamon is following my ['blog'], I have 2 followers now.
dcsignals is following my ['blog'], I have 3 followers now.
jerome-colley is following my ['blog'], I have 4 followers now.
mihserf is following my ['blog'], I have 5 followers now.
shaheer001 is following my ['blog'], I have 6 followers now.
helikopterben is following my ['blog'], I have 7 followers now.
carmasleeper is following my ['blog'], I have 8 followers now.
So if you liked this guide, and you'd like to see more, you know what to do.
Thank you for making awesome apps for Steem. I hope you have found my guide useful.
#steem #dev #steemit-dev #development #programming #python