Part 2: How To Stream And Filter The Blockchain Using Steem-Python

This tutorial is a continuation of Part 1: How To Configure The Steempy CLI Wallet And Upvote An Article With Steem-Python, so make sure to read this before starting this tutorial. Today we will learn how to stream the blockchain, what is actually broadcasted to the blockchain and how we can filter it to suit our needs!

What Will I Learn?

How to stream the blockchain
What posts on the blockchain look like
How to print information about a post
How to filter these posts

Requirements

Python3.6
steem-python

Difficulty

Basic

Tutorial

Streaming the blockchain

To access the blockchain and read data from it we can simply import the Blockchain class from steem-python and use its stream() function! This function takes a list of operations to filter for, so since we are interested in posts we can simply filter for that

from steem.blockchain import Blockchain
blockchain = Blockchain()
stream = blockchain.stream(filter_by=['comment'])
for post in stream:
    # DO SOMETHING

What do posts look like?

So, now we have a stream that we can iterate over with a for loop, but what do posts actually look like? Posts on the blockchain are in a JSON format. To find out what this looks like we can simply print them! We can use json.dumps() to print them in a valid JSON format, but before we do this, we need to create a converter() function that can handle objects of type datetime since they aren't serialisable

import json
import datetime

blockchain = Blockchain()
stream = blockchain.stream(filter_by=['comment'])

def converter(object_):
    if isinstance(object_, datetime.datetime):
        return object_.__str__()

for post in stream:
    print(json.dumps(post, default=converter))
    break

which outputs the example that can be found here. As you can see it doesn't include that much useful information. This is because we should map posts on the blockchain into a Post object!

from steem.post import Post
stream = map(Post, blockchain.stream(filter_by=['comment']))

Now that we've changed this we can run our program again, and it outputs something much more useful to us, as you can see in the example here. So much useful information!

Note: you can also use the post.export() function to get a post as JSON, but for the sake of this tutorial I have used json.dumps() as it makes it easier to print the JSON in a more readable format.

Printing information about a post

We now know what kind of information is contained in a post, so we can decide which part we want to print! For example, it would be cool to print the author's name, the title of their post and the tags they used. Looking at the the example we can see it has the keys "author", "title" and "tags". To print that we can simply use the following code in our for loop

author = post["author"]
title = post["title"]
tags = ", ".join(post["tags"])
print("{} posted {} with tags {}".format(author, title, tags))

which for example outputs

iseektruth posted  with tags

Obviously something has gone wrong, because when we looked at the example it clearly had an author, title and tags property, so why isn't it printing it? This is because posts on the blockchain can be either posts or comments, and comments don't have a title or tags! In the next section we will see how we can prevent this from happening!

Filtering the blockchain

To filter posts on the blockchain we can simply use an if statement! For example, if we want to make sure that the posts we print are actual posts and not comments we can use the is_main_post() function

for post in stream:
    if post.is_main_post():
        author = post["author"]
        title = post["title"]
        tags = ", ".join(post["tags"])
        print("{} posted {} with tags {}".format(author, title, tags))

which outputs the following

onegood posted The most beautiful period with tags life, life, photography, children

Great, it works! You can filter for anything you want! For example, if you want to only print posts with the tag utopian-io you can use the following code

for post in stream:
    tags = post["tags"]
    if post.is_main_post() and "utopian-io" in tags:
        author = post["author"]
        title = post["title"]
        tags = ", ".join(post["tags"])
        print("{} posted {} with tags {}".format(author, title, tags))

which outputs this for example

jestemkioskiem posted Moderation guidelines for Utopian's translation category. with tags utopian-io, utopian-io, utopian-io
tobias-g posted Indication of edited post with tags utopian-io, utopian-io, utopian-io, suggestion, busy
richardoccas posted Utopian-io – Bug report: The chosen language is not saved with tags utopian-io, utopian-io, bug, open-source, language, contribution
kad posted New Logo Design For Thomas with tags utopian-io, utopian-io, utopian-io, graphics, design, logo

Now you know how to filter posts on the blockchain! You could now try to filter posts by specific authors or the amount of votes for example, the options are limitless!

Note: by the time posts on the blockchain reach your program they could already be deleted, so you should make sure to check this. You can do this by putting everything in a while loop and your for loop in a try statement with an except clause, like so

while True:
    try:
        for post in stream:
            # DO SOMETHNIG
    except Exception as error:
        print(repr(error))
        continue

Curriculum

Credits

This tutorial was written by @amosbastian in conjunction with @juliank.

The code for this tutorial can be found on GitHub!

Posted on Utopian.io - Rewarding Open Source Contributors