Repository
https://github.com/steemit/steem
Introduction
There are many communities on the Steem network that use a different language than English. For example, I am actively involved in the #polish community using the @jacekw account. The purpose of the following analysis is to find the most active communities of this type, compare them and observe how they have changed over time.
Outline
- Scope of the analysis
- Tools
- Verification of initial data
- Different tags for the same languages
- Posts
- Payouts
- Average payout per post
- Authors
- Tags with prefixes
- Conclusions
- Proof of work
Scope of the analysis
The data has been downloaded from the SteemSQL database (tableComments) and refer to the first 6 months of 2018. The following script was used to download posts from the given tag.
SELECT url, total_payout_value, active_votes, json_metadata, created, body_language
FROM Comments (NOLOCK) c
WHERE depth = 0 AND
(CONTAINS(json_metadata, 'spanish') AND json_metadata LIKE '%"spanish"%') AND
YEAR(created) = 2018 AND
MONTH(created) <= 6
Tools
Verification of initial data
Potential language community tags have been manually selected based on 1000 most popular tags (from a 14 day time period).
lang_tags = [
'indonesia', 'spanish', 'aceh', 'kr', 'cervantes', 'cn', 'deutsch', 'castellano', 'venezuela',
'tr', 'polish', 'fr', 'myanmar', 'japanese', 'ru', 'pt', 'thai', 'ua',
'morocco', 'arab', 'pilipinas', 'steemit-austria', 'mexico', 'vn', 'rusteemteam', 'cesky', 'bangladesh',
'russian', 'hindi', 'br', 'arabic', 'teamserbia', 'steemromania', 'teamukraine', 'filipino', 'serbia'
]
There are different conventions here:
- country name e.g. #indonesia, #venezuela
- name of the language used, in English e.g. #spanish, #polish
- name of the language used, in the mother tongue e.g. #cervantes, #deutsch
- two-letter code e.g. #kr, #cn
- other name e.g. #steemit-austria, #rusteemteam
In my opinion, the name of the country is not the best choice for the language community tag, because this is the first tag that comes to mind if we want to add a post regarding given country in English.
The tags have been selected manually, so additional verification is needed, as some of them may not be relevant to language communities at all.
For this purpose, I used the column body_language from the tableComments. Below you will find charts showing the share of individual languages in given tags.
We can see that some tags are dominated by English. Such tags will be omitted from further analysis. Since we are not sure to what extent the body_language can be trusted, we will set the threshold quite low, at 30%.
| . | Tag | Ratio | Lang |
|---|---|---|---|
| 1 | castellano | 94.1 | es |
| 2 | br | 92.5 | pt |
| 3 | cervantes | 89.4 | es |
| 4 | kr | 87.4 | ko |
| 5 | thai | 87.3 | th |
| 6 | spanish | 87.0 | es |
| 7 | teamukraine | 85.5 | uk |
| 8 | venezuela | 84.7 | es |
| 9 | pt | 81.0 | pt |
| 10 | polish | 79.2 | pl |
| 11 | myanmar | 77.4 | my |
| 12 | tr | 68.7 | tr |
| 13 | japanese | 68.4 | ja |
| 14 | deutsch | 67.9 | de |
| 15 | fr | 65.1 | fr |
| 16 | steemit-austria | 63.1 | de |
| 17 | rusteemteam | 61.2 | ru |
| 18 | cesky | 59.7 | cs |
| 19 | indonesia | 53.2 | id |
| 20 | cn | 51.0 | zh |
| 21 | arabic | 50.6 | ar |
| 22 | ru | 48.3 | ru |
| 23 | mexico | 47.4 | es |
| 24 | aceh | 46.2 | id |
| 25 | morocco | 45.6 | ar |
| 26 | arab | 41.2 | ar |
| 27 | ua | 34.6 | uk |
| 28 | pilipinas | 33.1 | tl |
| 29 | russian | 32.9 | ru |
| 30 | hindi | 30.1 | hi |
| - | - | - | - |
| 31 | filipino | 19.7 | tl |
| 32 | vn | 17.8 | vi |
| 33 | serbia | 15.9 | sr |
| 34 | bangladesh | 15.4 | bn |
| 35 | teamserbia | 15.0 | sr |
| 36 | steemromania | 13.9 | ro |
The last 6 tags will be deleted.
Different tags for the same languages
It also appears that some communities use the same language as others. Let's see what the relations are between them, whether they have a large common part.
es : castellano, cervantes, spanish, venezuela, mexico
pt : br, pt
uk : teamukraine, ua
de : deutsch, steemit-austria
ru : rusteemteam, ru, russian
id : indonesia, aceh
ar : arabic, morocco, arab
def plot_venn(tags):
plt.figure(figsize=(6, 6))
fn = venn2 if len(tags) == 2 else venn3
fn([tag_urls_dict[tag] for tag in tags], map(lambda t: '#' + t, tags))
plt.show()
for lang_code, tags in same_lang_dict.items():
if 2 <= len(tags) <= 3:
plot_venn(tags)
plot_venn(['spanish', 'cervantes', 'castellano'])
plot_venn(['spanish', 'mexico', 'venezuela'])
pt : #br, #pt
uk : #teamukraine, #ua
de : #deutsch, #steemit-austria
ru : #rusteemteam, #ru, #russian
id : #indonesia, #aceh
ar : #arabic, #morocco, #arab
es: #spanish, #cervantes, #castellano
es : #spanish, #mexico, #venezuela
The #teamukraine tag is practically completely contained in #ua, so it is going to be omitted from further analysis, especially that both of them concern the Ukrainian community. For #pt and #br the situation is similar, but #br seems to be a separate (Brazilian) community.
Posts
Let's see how many posts were added in each tag.
| . | Tag | Posts |
|---|---|---|
| 1 | indonesia | 300522 |
| 2 | spanish | 282685 |
| 3 | aceh | 258482 |
| 4 | kr | 242179 |
| 5 | cervantes | 158545 |
| 6 | cn | 74577 |
| 7 | deutsch | 71291 |
| 8 | castellano | 61437 |
| 9 | venezuela | 58809 |
| 10 | tr | 57838 |
| 11 | polish | 29744 |
| 12 | myanmar | 20385 |
| 13 | fr | 20350 |
| 14 | japanese | 19592 |
| 15 | ru | 18445 |
| 16 | pt | 16875 |
| 17 | thai | 16681 |
| 18 | ua | 14414 |
| 19 | morocco | 8851 |
| 20 | arab | 6444 |
| 21 | pilipinas | 6389 |
| 22 | steemit-austria | 6386 |
| 23 | mexico | 4643 |
| 24 | rusteemteam | 4615 |
| 25 | cesky | 3956 |
| 26 | russian | 2432 |
| 27 | teamukraine | 2133 |
| 28 | hindi | 1990 |
| 29 | br | 1326 |
| 30 | arabic | 1240 |
We can see a great diversity here, communities from leading places have several hundred times more posts than those from the end. The Spanish language tags are top of the list, which may indicate that Steem is well known in South America.
Author rewards
However, the number of posts is an insufficient indicator, because it is very easy to spam a given tag with low value posts, and this will not indicate the popularity of a given tag at all. Therefore, let's also look at the sum of rewards in individual tags.
| . | Tag | Author rewards |
|---|---|---|
| 1 | kr | 342070 |
| 2 | spanish | 166190 |
| 3 | cn | 158431 |
| 4 | cervantes | 105259 |
| 5 | deutsch | 86543 |
| 6 | indonesia | 81085 |
| 7 | tr | 63973 |
| 8 | aceh | 42160 |
| 9 | fr | 36596 |
| 10 | castellano | 35293 |
| 11 | japanese | 26803 |
| 12 | pt | 22386 |
| 13 | myanmar | 19214 |
| 14 | polish | 19053 |
| 15 | ru | 18731 |
| 16 | venezuela | 17815 |
| 17 | steemit-austria | 13182 |
| 18 | ua | 12459 |
| 19 | thai | 12041 |
| 20 | morocco | 9888 |
| 21 | mexico | 8887 |
| 22 | arab | 6835 |
| 23 | br | 4579 |
| 24 | cesky | 4246 |
| 25 | pilipinas | 3428 |
| 26 | rusteemteam | 2364 |
| 27 | hindi | 1325 |
| 28 | russian | 718 |
| 29 | teamukraine | 519 |
| 30 | arabic | 414 |
The table looks quite similar to the previous one. It is also worth to look at the average rewards in a given tag. This will allow us to find out how rich the community is.
Average payout per post
| . | Tag | Average author rewards per post |
|---|---|---|
| 1 | br | 3.453 |
| 2 | cn | 2.124 |
| 3 | steemit-austria | 2.064 |
| 4 | mexico | 1.914 |
| 5 | fr | 1.798 |
| 6 | kr | 1.412 |
| 7 | japanese | 1.368 |
| 8 | pt | 1.327 |
| 9 | deutsch | 1.214 |
| 10 | morocco | 1.117 |
| 11 | tr | 1.106 |
| 12 | cesky | 1.073 |
| 13 | arab | 1.061 |
| 14 | ru | 1.016 |
| 15 | myanmar | 0.943 |
| 16 | ua | 0.864 |
| 17 | thai | 0.722 |
| 18 | hindi | 0.666 |
| 19 | cervantes | 0.664 |
| 20 | polish | 0.641 |
| 21 | spanish | 0.588 |
| 22 | castellano | 0.574 |
| 23 | pilipinas | 0.537 |
| 24 | rusteemteam | 0.512 |
| 25 | arabic | 0.334 |
| 26 | venezuela | 0.303 |
| 27 | russian | 0.295 |
| 28 | indonesia | 0.270 |
| 29 | teamukraine | 0.243 |
| 30 | aceh | 0.163 |
We can see that the differences are significant. It should also be taken into account, however, that tags with a small number of posts may not have reliable results.
Authors
We can look at the popularity of a given tag from a different angle - looking at the number of authors, not posts. This indicator seems even better, because it omits situations where a given user spams a very large number of posts.
| . | Tag | Authors |
|---|---|---|
| 1 | spanish | 48312 |
| 2 | indonesia | 42307 |
| 3 | aceh | 40846 |
| 4 | cervantes | 31520 |
| 5 | kr | 25622 |
| 6 | venezuela | 18329 |
| 7 | castellano | 15121 |
| 8 | cn | 9226 |
| 9 | deutsch | 9109 |
| 10 | tr | 6931 |
| 11 | polish | 4785 |
| 12 | japanese | 3118 |
| 13 | fr | 2976 |
| 14 | myanmar | 2851 |
| 15 | ru | 2083 |
| 16 | thai | 2060 |
| 17 | mexico | 1998 |
| 18 | pt | 1829 |
| 19 | arab | 1058 |
| 20 | morocco | 1039 |
| 21 | steemit-austria | 880 |
| 22 | ua | 854 |
| 23 | pilipinas | 846 |
| 24 | russian | 796 |
| 25 | hindi | 638 |
| 26 | rusteemteam | 458 |
| 27 | arabic | 441 |
| 28 | cesky | 369 |
| 29 | br | 291 |
| 30 | teamukraine | 81 |
Now we see how small some communities are! And if a community does not exceed a certain threshold, its members may prefer to post in English, which makes the growth of such a community more difficult.
Tags with prefixes
A common problem in language communities (I say this as someone who is actively involved in the #polish community) is how to use other tags. If I make a post in Polish and tag it: #polish #bitcoin #cryptocurrency, these last two tags will make the post reach also English-speaking audiences, so it is not a good solution.
Another idea seems to be to use tags in the mother tongue, but then you may find tags that function as a word in both languages, e.g. #film. This shows that it is not a good solution either. Therefore, the Polish community decided (based on the convention adopted by the Korean community) on the convention of tags with the prefix pl-. This makes it possible to separate posts in Polish from the English-speaking audience.
Let us therefore look at which other communities use such a convention. The table below shows how many unique tags are with the specified prefix.
| . | Prefix | Count |
|---|---|---|
| 1 | kr | 2302 |
| 2 | pl | 815 |
| 3 | cn | 221 |
| 4 | ru | 168 |
| 5 | jp | 64 |
| 6 | de | 31 |
| 7 | tr | 24 |
| 8 | pt | 12 |
| 9 | fr | 8 |
| 10 | ua | 5 |
| 11 | es | 5 |
We can see that the Korean community has the largest number of such unique tags, followed by the Polish and Chinese communities. As far as the Polish community is concerned we can see a tree of these tags on the website: https://steemweb.pl/categories (by @rafalski)
Let's see what are the most popular tags of this type.
| . | Prefix-tag | Count |
|---|---|---|
| 1 | #kr-newbie | 83291 |
| 2 | #kr-life | 23164 |
| 3 | #kr-writing | 19077 |
| 4 | #kr-event | 13164 |
| 5 | #cn-reader | 13133 |
| 6 | #kr-daily | 12263 |
| 7 | #kr-art | 8748 |
| 8 | #kr-travel | 7077 |
| 9 | #kr-food | 6037 |
| 10 | #pl-artykuly | 5987 |
| 11 | #kr-pen | 5678 |
| 12 | #kr-coin | 5494 |
| 13 | #kr-join | 5484 |
| 14 | #kr-news | 5011 |
| 15 | #jp-newbie | 4171 |
| 16 | #kr-overseas | 4055 |
| 17 | #kr-diary | 3812 |
| 18 | #kr-gazua | 3752 |
| 19 | #kr-series | 2980 |
| 20 | #kr-funfun | 2738 |
| 21 | #kr-dev | 2665 |
| 22 | #kr-book | 2601 |
| 23 | #kr-youth | 2563 |
| 24 | #kr-story | 2355 |
| 25 | #cn-malaysia | 1957 |
| 26 | #kr-economy | 1777 |
| 27 | #kr-hobby | 1708 |
| 28 | #kr-steemit | 1706 |
| 29 | #kr-game | 1693 |
| 30 | #pl-fotografia | 1612 |
As we can see, most of the tags are Korean.
Conclusions
We have managed to find 30 language communities that do not use English. There are probably more, but the process of finding them is not a simple one, because of the different tagging conventions that have been adopted. All these communities are probably waiting for the appearance of the functionality Communities, because this will make it easier for them to function.
It also appears that the size of language communities is not always correlated with the number of people using a given language / country size. An example of this is tag #kr, which is high in all lists and has a relatively small population compared to e.g. the Chinese community.
Some communities use almost only their mother tongue, while others have chosen to use English (e.g. #serbia). It is also worth remembering that the Russian community decided at some point to move to its own blockchain: https://golos.io, but on the other hand, this did not make it completely disappear from the Steem network. Interestingly, there are also quite a big countries that do not have a language community.
There are many other issues that could be explored here, such as how the community formation process looks like, who the pioneers are, who the leaders are (if any).