( screencap from https://steemit.chat/channel/witness-blocks )
Laid down for a bit of a nap earlier and woke up to my main witness server overflowing it's shared memory file... Causing me to miss nearly as many blocks as I'd missed in over a year of running a witness campaign. My apologies for causing any lag on the network!
Of course my witness failover script shit itself and was unable to switch over my witness keys to my backup (which thankfully was running 16GB shared memory file) and thus I missed 77 blocks.. Unacceptable, will be getting my failover script working before I sleep again. Serves me right for sleeping! What they don't tell you about being a witness is that you are ALWAYS on call, given had I not been napping I'd been able to rectify this issue earlier..
( screencap from https://steemdb.com/witness/misses )
Dropped from rank 7 to 12 it looks like, which is to be expected..! Generally speaking if a top 20 witness is failing at his block producing job it's up to the community to put him in a backup position, no hard feelings towards anyone who unvoted my witness during that overflow blunder! It's our duty to keep the networks blocks flowing I say!
Setting Your Shared Memory File Size
By default steemd sets the shared memory file to 32-56GB so for most this won't be an issue. However some witness server operators like myself choose to run their shared memory file in the systems RAM as shown in @abit's post Best Practice Running steemd v0.16.0 which causes the witness server to overflow with a 12GB shared memory file size set in the config.
config.ini file is located in the witness_node_data_dir directory.
Modify the shared-file-size parameter value to 16GB or more to ensure that you don't end up overflowing and missing blocks! While this will suffice for now it's almost guaranteed that in the future this shared memory memory file size may need to be increased!
I'm off to go write a NodeJS failover script to prevent this from happening again. Was previously using @jesta's failove script seen here but it seems either user error or outdated dependencies may have caused it to fail. I'll open source my failover script once completed.
Have a gooder everyone!
 | VOTEfor Witness! | |
|---|