Version 0.8 took a while, this article will explain why. But it comes packed in with a lot of new things that were highly requested. Almost a year since our inception, we are now hitting the 8th iteration of DTube, and how far we've gone. This update once again introduces more new features to make the experience more complete and similar to the mainstream video platforms. But before talking about new features, I have a big announcement:
DTube is now fully open-sourced
Up until now, everything was open-source, except the main application which was only available in minified format, which was apparently very frustrating for a lot of people.
I'm glad to announce that DTube is now a 100% free open-source software. Our main application is now is available for all to read or modify on github.com/dtube/dtube. This is another move in the hope of improving the transparency of our project and allow contributions from anyone.
Notifications
I've talked about notifications a lot in the past articles, and it is something I thought we would have earlier in our development. Finally, here it is. It required a bunch of changes in DTube so that any content could be viewable in DTube, including non-video content.
Our notifications are totally similar to what busy.org implemented, only the client-side is slightly different and should allow you to better identify which notifications you have clicked on already.
DTV: Livestreaming now available on DTube
A big new feature, that I am sure everyone will enjoy, is livestreaming. We are using MistServer, a centralized livestreaming server, with an open-source version.
On top of Mist, I built an api to safely generate streaming keys that are associated directly to a steem username. When you click the button to get your stream key, it will create a custom dtubeStreamVerif transaction in the background that will verify your steem identity and authorize your stream token. This means you can start livestreaming, without necessarily having to publish a new root post on the blockchain.
Of course, this feature is indeed monetized. Once streaming, just publish your stream on the 'Go Live' page, and a new steem content will be created. Your livestream will then be displayed on the DTV section of DTube. Just like for videos, as you keep getting more and more upvotes, your stream will go up in the listings and grow your audience organically.
While decentralizing livestream would be good, it's a hard task that will take time. It is our end goal. However, unlike mass video encoding, livestreaming servers is a very open market with a lot of alternatives, including open source ones. We decided that it is a good choice to use just that for now, as it doesn't create much cost or maintenance time for us, while fulfilling the biggest demand of our community.
I wanted to release this as just a basic streaming feature, but many private users complained that a livestream in itself would serve no purpose if the video wasn't available later. We are therefore storing all of your livestreams inside our infrastructure, and clicking on an old content will show up the video matching the time of the publication on the steem blockchain. These videos, as well as the livestream feeds, are both unencoded (a bit how our videos were unencoded in earlier versions of the project), so keep that in mind, and make sure that your stream quality isn't too high if you want users with bad internet to be able to follow your stream.
In the next MistServer release, we should also have access to WebRTC broadcasting which will enable direct in-browser streaming through WebRTC, i.e. no more need for software like OBS or Larix Broadcaster to start streaming.
DTalk: Steem Private Messaging
Steem Private Messaging is a subject that seems to come back into life once in a while, without ever finding a working solution. I personally believe that if all top social networks have implemented messaging, there is a reason. Currently, most steemians are using steem.chat or Discord. These solutions are poor because they require users to verify who they are speaking with, if they don't want to get hustled. If we want a true way to message between steem users, we shouldn't need a secondary account on Discord on Steem.chat.
Some other developers have tried to come up with solutions to this issue before, such as Mangosteem by @mynameisbrian or Steem Messenger by @therealwolf. These solutions failed to get adopted by the community because they are on-chain. Mangosteem was spamming the chain as comments and became unusable after a short time. Steem Messenger is using transfer operations to send (short) messages, so each message will cost 0.001 STEEM or SBD. Also, even if you wouldn't be able to decode a message, you'd still know who is speaking with who.
After carefully considering these facts, and badly enough wanting to allow DTube users to send themselves messages, I've tried coming up with solutions. A simple centralized solution would be using a modified irc server that can can login STEEM users by asking them to decode a message only them can decrypt. This solution works fine, and should scale as good as a traditional IRC server (pretty good).
But, can we decentralize it?
If we can do it for pictures and videos, why not text strings? Because if we do not want to use the STEEM blockchain as a database like we usually do, storing messages would serve no purpose as we would have no way to fetch all messages from the same discussion easily... What we need is a decentralized database that runs inside the browser, inside the DTube app...
Yes we can!
For this purpose, DTube is now using Gun, a decentralized graph database technology. It's created by @marknadal, with whom I had the luck of collaborating to make DTalk reality. A good example of gun being used in production today is notabug.io, a decentralized reddit, running 100% on gun, no blockchain, no backend. DTube is now another big example of the capabilities of Gun.
Just like for IPFS, DTube will have a Gun node running 24/7 inside the infrastructure, which should help correspondants seeing new messages even if the sender disconnected, and keep your encrypted messages stored somewhere even if you wipe all your browsers.
All the messages are encrypted end-to-end, and finding out who speaks with who is impossible if you cannot decrypt the message. The only caveat, is that you will need to publish your gun key inside your profile in order to be discoverable and receive messages from other users. You should see a red button on the DTalk part of DTube. It's a one-time thing to do, except if you reset your STEEM keys, in which case you will need to do it again and all your old conversations will be lost.
So while the first option would most likely have worked better and with less work, the second option is our final goal, that's why we are starting this feature directly with a decentralized approach, even if it might be slightly buggy in the first weeks while we handle all the details.
Handling DMCAs
Not really a good feature, but we have started blocking a few videos from the d.tube domain based on DMCA complaints we have received. We are now running a super slim 18-lines server that identifies DMCA'd videos.
This is unfortunate, but all of the requests received so far have been 100% legit, mostly reposted movies or links to pirated software. A large majority of our users are playing fair and uploading only original content, let's keep it that way!
Clustering our infrastructure
After the previous update, we were managing about 10 different size specialized servers for D.Tube. Some were focusing on storage (storage servers), some on computational power (encoding servers), some were for general purpose things or testing. This was convenient and simple to manage while we had a low number of server, but the growth of the project required us to professionalize that part, even though in the far future we aren't supposed to run anything server-side, it isn't the case currently.
Clustering everything, is a bit like creating one big machine by merging all other machines together. This big machine is what we call a cluster. Then with virtualization tools like Docker, we are able to launch many small instances of the same software inside the cluster, without having to rent a new server and wait everytime.
Dockerification of all our software
These small instances are all ran through Docker, a popular way to achieve this. Most STEEM witnesses are also using docker for running the STEEM blockchain.
Most of the software we run server-side is our own code, and the ipfs-uploader for example wasn't quite ready for cluster support. We had to adapt everything so it would be easy to start/stop any number of the same DTube service and scale up.
At the moment, our cluster runs a total of 7 services:
- traefik: the service that handles all the traffic and reroutes it to the good docker container
- IPFS Uploader: our upload service (ffmpeg encoding, snap resizing, ipfs node)
- Streaming: MistServer combined with our own stream keys api.
- Direct video delivery (more on that later)
- GunDB node
- MariaDB (our small sql database for accounting and @dtube.rewards)
- vip.d.tube: a small website for quickly creating steem accounts if you have a code
Data storage and redundancy
Another big issue of our old setup, was what happened when a drive died. All the data inside it died with it, and unless the file was copied on another ipfs node, or the author reuploads the same file, the video would never load again. Managing redundancy with IPFS was the original plan, but using ipfs pin lson large datastores like ours takes far too long and isn't manageable.
Instead, we tried to use existing enterprise-grade solutions to basically create a huge cloud folder for all our videos, and every bit of data inside this folder, gets written on at least 2 different real physical machines. We first tried 'GlusterFS', which ended up being really slow because of all the small files created inside the IPFS datastore. We are finally using Ceph which seems to do what we want it to do.
Migration
An important part of the cluster project, was to move everything from our simple setup, into the new thing which had no prior testing. We did a few testing days where half of our uploads would go on the cluster, and half on the old setup. We finished migrating the ipfs uploading and video delivery into the cluster on June 19th.
IPFS getting slow, adding direct video delivery
In the middle of all this change, the IPFS network gateway started getting slower to load videos, even when using our old-school way of doing things. Users were complaining. We had to do something. We built a new service running on the 'video.dtube.top' domain, which serves videos straight from our cluster, exactly like IPFS, but without going through the IPFS network. Files are still kept inside the IPFS datastore and fetching the video from IPFS still works.
While this was true 1 month ago, it seems IPFS manages to load videos quite quick again. Wait & see
Issues we had
All these changes in the infrastructure came at quite a heavy cost. Our service has been unstable at times during these past months.
We had a big incident on July 15th which caused a ~6 hours downtime of the uploads and video delivery.
We also had multiple short-term incidents, mostly during the migration period. Either super long queues for the upload, or videos not playing.
On behalf of the DTube team, I'd like to apologize for all these failures. Our service has been running uninterupted for the past 3 weeks now, and we think it will be good now. In case of issues, please keep contacting us in the #upload-issues channel like you've done in the past, this greatly helps us.
Get started on DTube development with Utopian.io
While I am only announcing today on our blog about the fact that the main DTube app is open-source, we already collected 75 stars and many pull requests from contributors, mostly coming from Utopian.
Contributing to DTube's development with Utopian is a unique possibility today for open-source developers, and we already had multiple positive experiences of developers doing work for DTube, out of their own motivation, without expecting a payout. Except they end up with crypto on their account, and a lot of visibility for their work.
The process is easy:
- Find something to work on. Either in our tagged GitHub issues or by reaching out to me directly on discord.gg/dtube
- Write good code and submit it as a pull request on GitHub.
- We will check your pull request and merge it, or comment on the issues we discovered.
- Within the next 14 days of the merge, submit a post on the STEEM network. It can be a DTube video.
- The first tag must be
utopian-io - The second tag must be
development.
- The first tag must be
- ???
- Profit and become famous
DTube 0.8 Open Source Contributions
New Curation Style
Contributor: @wehmoen | Utopian post | New Repo
Since a couple of weeks now, we have a new discord bot that helps our curators do their work more precisely.
Instead of simply voting always at the same percentage (20%), curators can now control the voting weight of the bot through emojis! Each reaction (👍,👎,💯,🎲 or ❤) has a different impact on the vote.
DTube Onboarding Platform: vip.d.tube
Contributor: @wehmoen | Utopian post | New Repo
It's something that has been requested to me many times, and that I didn't want to do before, but it's getting important now. We will now be able to generate voucher codes that enable people to get an instant STEEM account. We plan to distribute these codes at events, probably directly on our business cards as a forst. We can also give codes to popular Youtubers wanting to make the switch without having to wait in the future.
Small UI and graphical lifting (menu, upload, homepage, etc)
Contributor: @happyrobot | PR
0.8 looks better than 0.7, and all of that is thanks to our new contributor @happyrobot. Most noticeable changes will be in sidebar and on the upload page, but almost everything got a small lifting.
Bottom Drawer for languages and notifications lists
Contributor: @happyrobot | PR
Our language selection menu was getting quite long, even on desktop, making languages at the end of the alphabet get out of the screen. Notifications were also running into this issue on mobile where it was impossible to display a floating menu with all notifications inside. Therefore we created a new way to display long list of informations in DTube.
Remembering user settings for nightmode and voting weight.
Contributor: @kirkins | Utopian post | PR
Another small annoyance when using DTube was how your voting weight or nightmode settings was reset when you opened a new tab or refreshed the current one. This is now working as expected and will be remembered.
Final words
This update is another big step towards the completion of a large project. Last year when I originally published about the 0.1 version of DTube, I had honestly no idea where things were going to go, I wasn't even sure it would work.
I want DTube to go away from the proof-of-concept stage and turn into a fully-integrated product. Right now the machinery isn't very polished, but things are getting plugged in at a quick rate.
There are still tons of features which will be needed to make DTube the video platform that producers and consumers equally want to use in the future. Playlists, VOD, Twitch-style donations, analytics, ads platform, I keep coming up with good feature ideas as the project goes along.