Python steem API not fast enough to keep up with new blocks

I've been having some issues with my @croupierbot bot recently. At first, everything seemed to be working using the stream_comments method in a loop. Well, working? I needed to run everything in a while loop with a try/except, catching steembase.exceptions.PostDoesNotExist exceptions and restarting the stream_comments loop all the time, but after that, my beta tests did OK. For a while that is.

Then, after @bigpanda tried to use my bot, I noticed something was off. @croupierbot missed many of the times its name got mentioned in the stream. It looked like this was getting worse over time in fact. And apart from that, the time between calling in @croupierbot and getting the first response often went up as the bot was left running for a larger amount of time.

So I thought, hey, maybe it is the exception stuff, maybe the restarting of the stream API had me skipping stuff, let's write to the get_blocks_range API instead.

The results weren't good. No more exceptions to catch, but the timelapse problem went completely off the scale. So what to do. I stripped down my script and removed all the stuff that really did something. I ended up with a little script that tried to keep up with new blocks being added to the blockchain. Only that, nothing more. And guess what? The script couldn't even keep up with doing nothing. That is, it seems the Python steem API is too slow to allow for writing a bot. Silly, right? I must be something wrong here, right? Well, I thought so, but after looking at my script for hours, and I really can't see anything stupid. Here is my code:

And here is the result of letting the code run for a number of minutes:

Messed up, right? In the time it takes to request ten blocks from the steem API, more often than not more than ten new blocks get added to the blockchain! Am I doing something really stupid here that isn't obvious to me right now? Or has the steemit network become too busy for the API to keep up with new block creation?

For now, it seems I will need to find a different strategy for @resteembot. I hate the idea of asking for transactions, even if they are $ 0.001 SBD rather than scraping for mentions as I was doing, but it is possible that currently, it is my only option to get @croupierbot to work relyably. If you spot anything stupid in my code or know a better coding strategy for scrapping for mentions, please let me know in the comments.

Tnx.

H2
H3
H4
3 columns
2 columns
1 column
11 Comments