SteemData 1.2
I've decided to ship early, and not wait until SteemData 2.0. The main reason is that I'd like to push out all the breaking changes now, to reduce the amount of pain in the future.
Features in 1.2
Fast updates and eventual consistency
Before 1.2, I would run a handful of workers in a loop, and scrape account related updates one by one. Steem now has over 120,000 accounts, and this approach certainly doesn't scale. It also means that an account can only be updated once every few hours, and thus some of the data is stale.
I have solved this problem by switching to an asynchronous event based model (powered by Celery and RabbitMQ, the distributed queue), where posts, accounts and their virtual operations are updated shortly after new blocks become available.
I have repurposed the old worker model as a fail-safe - if for whatever reason the event based approach fails in such a way that it would cause loss of data - the background worker will back-fill the missing data afterwards.
Structural Changes and Types
This release contains a handful of design improvements and changes, which are not backwards compatible. I do not expect any major breaking changes for 2.0.
Also, the typing support has been improved greatly.
Historic Prices
I've added hourly snapshots for STEEM, implied SBD and Bitcoin prices.
Performance Improvements
The new Mongo deployment is wriredTiger enabled.
I have reworked indexes on all collections, which yields in over 2-10 fold query performance improvement for most historic queries.
SteemData is now also hosted on a more expensive, Intel i7 6700k powered server with 64GB RAM. The hardware upgrade should yield over 2x performance gain.
Open Source
All of the code powering SteemData is now available on Github, and is licensed under highly permissive MIT.
steemdata-node
If you're looking for a Docker based, easy to use steemd RPC deployment, this is it.
It comes with all blockchain plugins enabled, latest seed node list and automatic blockchain snapshot download on first run for quick syncing times (thanks to @gtg).
steemdata-mongo
This repo contains all the code that is responsible for syncing STEEM blockchain with MongoDB.
steemdata
This is a core library for working with STEEM blockchain data. It is database agnostic (could be used for SQL or any other database in the future).
steemdata.com
Right now, the website only hosts basic instructions and stats.
Eventually, I would like to build:
- an API for 3rd party apps
- blockchain explorer
- steemle inspired charts and analytics
TODO (until next release)
- Integrate Comments
- Add Relationships via HRefs
- Create Sample Notebooks
- Documentation!
Now that the stable base is in place, I'd like to work on making this project more useful and friendly to people who can benefit from it. If you're a developer, please talk to me (I am @furion on steemit.chat)
Upgrade Now
The old version of SteemData will be shutting down on Feb 10th. Please upgrade to SteemData 1.2, see steemdata.com for connection info.
Crowdfunding
We have raised $5,120 of the $5,000 goal so far. Big thanks to @cass for making this project possible.
| Supporters | |
|---|---|
| @cass | $4,900 |
| @fabien | $100 |
| @abit | $100 |
| @tuck-fheman | $20 |
The donations should be sent to @steemdata, and the list of friendly donors will be published and updated here, as well as in future announcements.
If you'd like to support my work, feel free to vote @furion for witness.