Research Study: @sockington is more influential than @chrisbrogan

This Webecology research report has been making the rounds on Twitter. I haven’t had time to read it until now, here are my reading notes:

The Webecology team uses large scale data mining to identify patterns indicative of online culture and community. Wish I’d do this, too – and will, as soon as I find a research partner to help with the data mining part.

For this project, the authors set out to create a more accurate measure of influence on Twitter that goes beyond either:

1. number of followers; or
2. followers/friends ratio

The authors defined influence on Twitter as:

influence on Twitter = the potential of an action of a user to initiate a further action by another user

Specifically, influence means the potential of a tweet to generate replies, mentions (conversational behaviors), RTs, and attributions (content-pushing behaviors).

This is an atheoretical, operational definition of influence (the study’s Achille’s heel).

As far as I understand, all 4 actions were weighed equally. So, a RT factors the same as an @reply in determining influence.

They selected 12 Twitter accounts to study. The selection was based on this criterion: the 12 accounts were  ”widely perceived to be among the more influential users on Twitter.” It is not clear who did the perceiving, and what definition or measure of influence they used in the process of perception. IMO, the arbitrary selection of the sample is another major weakness – but in this case, I can live with it, because the purpose is not to derive conclusions about Twitter culture as much as it is to demonstrate how the methodology can be used.

Then, the 12 users were grouped into 3 categories. Here is a table with the accounts they analyzed, and their number of tweets over 10 days, as well as the number of followers and friends at the end of the 10 days:

—

Celebrities	Username	Tweets	Followers	Followees
Ashton Kutcher	aplusk	3,205	3,407,385	209
Shaquille O’Neil	THE_REAL_SHAQ	2,072	2,092,541	562
Stanley Kirk Burrell	MCHammer	6,016	1,331,797	31,202
Sockington	sockington	5,711	1,089,984	380
Justine Ezarik	ijustine	7,718	605,441	3,039
News Outlets	Username	Tweets	Followers	Followees
CNN Breaking News	cnnbrk	1,096	2,712,530	18
BarackObama.com	BarackObama	330	2,018,016	761,851
Mashable.com	mashable	17,914	1,363,510	1,925
CNN	cnn	11,607	193,625	50
Social Media Analysts	Username	Tweets	Followers	Followees
Gary Vaynerchuk	garyvee	7,532	862,790	9,683
Chris Brogan	chrisbrogan	48,341	94,715	88,431
Robert Scoble	Scobleizer	23,112	94,295	2,423

The data that they mined was as collected over 10 days, in August 2009. The data included:

• The 2143 tweets generated by the 12 users
• The 90,130 actions (responses, RTs) triggered by the original 2143 tweets
• All the tweets generated in connection with the 12 users (by their followers and friends;a total of 134, 654 tweets, 15,866,629 followers, and 899,773 friends/followees)

The authors produced 2 types of influence reports, based on the type of action that was triggered:

1. conversational action (people replied, or mentioned the user – e.g. “meeting @stockington for catnip”)
2. content-pushing action (people retweeted, or gave attribution – e.g. “via@username”)

Please note that a mention may or may not be a response to a tweet. If they were not responses to a tweet, they fall outside the authors’ definition of Twitter influence, and they should have been excluded from the analysis.

Here we go, on to the findings:

Conversational action

This graph shows you the amount of conversational activity (@replies and mentions) each user got in response to one (average) tweet.

Content action

This graph shows you how much content action (retweets and attributions) each user got for each (average) tweet:

So here we see that, per tweet, @sockington did get more retweets than @chrisbrogan.

The authors claim that these graphs of influence/tweet are the most accurate measure of Twitter influence so far. Therefore:

@sockington IS more influential on Twitter than @chrisbrogan,

because the fake cat gets more retweets. (sorry, @sockington, I do love you!!!)

I know exactly what you’re thinking, it starts with B and ends with T.

That’s because here we have a problem of construct validity. The measures do not actually measure influence. I wish the authors had read some research in communication & persuasion about the concept of influence, then worked their way from a conceptual to an operational definition.

Obviously, @sockington gets more retweets because he’s cuter & funnier than @chrisbrogan (sorry, Chris!). We don’t know why people reply or retweet. This study ignores a very important aspect of human relations: meaning. There is meaning in tweets, and meaning in why people retweet. But that is not captured in this study.

That being said, the report shows what can be done with data mining – it’s awesome! With a bit of help from people who know how to study meaning (hint, hint!), this type of research will be extremely valuable.

If anything, let this be an argument for computers & communication people working together, across disciplines.

In a future post, I will review conceptual and operational definitions of influence.

How to read a research article

Most research articles you find in academic journal follow a similar recipe. If you understand how the article is structured and what to look for in each section, you can read articles much faster. I can get what I want from a research article in 5 minutes or less. When I started grad. school it took me 45-60 minutes to get through a research article and I still didn’t get much out of it. I wish someone had taught me how to read them.

Here are my lessons, based on my experiences. They work for me. I hope they work for you, too. If they don’t, use this as a starting point to figure out your own reading process.

Understanding the anatomy of a research article will also help you write easier.

Title

Usually long and cryptic. Most titles are poorly written. I don’t pay much attention to the title.

Abstract

I read it carefully and look for:

• purpose of study/research question
• a hint as to research methods
• key results

Introduction

I read the introduction looking for the following information:

• explanation of the problem the study addresses
• explanation of the larger context of the problem
• argument about the importance/need/relevance of studying the problem
• purpose of the study
• an overview of how the article is structured, and how the next section is organized

Literature review

It may be called something else, or the article may not even have headings – but it should be there somewhere. The literature review should accomplish 2 purposes:

1. make an argument for the need to conduct this specific study (identify a gap, or a need in previous literature)
2. present the previous theories, concepts, etc. that this study uses and builds upon

Usually, each paragraph or small section of the literature review covers a body of literature (the best lit. reviews are organized thematically, IMO). When reading the literature review it is important to identify these major themes. They give you a lay of the land.

Imagine the body of literature is a garden. The article you’re reading attempts to plant a new seed in this garden. Before doing so, the authors explain the layout of the garden (vegetables here, flowers there, weeds over there) and they explain why their plant is needed and where it fits in.

When reading the lit. review, you get a feel for this garden. If you are:

• very familiar with the literature, the lit. review confirms that the authors looked in all the right places and didn’t reinvent the wheel. OK to skim.
• completely unfamiliar with the literature, this section will be terribly confusing. Don’t worry. All you have to get out of it are the major themes (sections of the garden). You can come back later and examine each individual plant. OK to skim.
• are trying to learn the literature – read carefully, and mark on the list of references the sources you want to read.

The literature review ends with the research question(s). Find them and highlight them. They are promises that the article should deliver on.

Methods

This section explains the research methods and procedures used for the research study. Read them carefully, make sure they are valid. If the research methods are faulty, the data are not to be trusted. If the research methods are absurdly faulty, stop reading here. Go back to the literature review and the list of references and see if they can help you find better articles on the topic.

Results

In this section, the authors present their data, along with their (statistical or interpretive, etc.) analysis. This is as close as you can get to the raw data. This section, in a quantitative article, should be as free as possible of interpretation. Try your best to understand the results for yourself, so you can create your own interpretation of what they mean. But, if the statistics baffle you AND if you trust the authors, skim this section and move on to:

Discussion

This section explains what the results mean, in the context of the garden (literature review). You should see how the problem from the introduction is solved, how the research questions are answered, and whether the purpose of the study was accomplished. I usually read this section very carefully, because it tells me what the authors think they have accomplished.

Either here or at the end of the conclusion, you will find suggestions for future research. These can be very useful for your own literature review – you can cite the article, if it calls for exactly the research you’re doing. You can use this to support your own argument about the need for your research.

Conclusion

The first part of the conclusion should be a summary of the entire paper. I read it carefully, because the repetition helps me remember what I read. The last part of the conclusion is usually the most difficult part to write, very often fluff, and I don’t feel guilty about skimming or skipping it.

I used to teach this recipe to graduate students and they found it very helpful. I hope you do, too. Please share your own reading and writing tips, and ask me other questions you may have about graduate school.

There are several books that can help you, and the APA style manual has a chapter that explains the structure of APA research papers.

[update:] Barbara Nixon created a slide presentation for this content:

Quantum Physics

Someday, I will understand quantum physics. But since in the past few weeks I’ve been unpacking, unpacking, unpacking, unpacking, unpacking… (you get it)… OK, never mind. Here’s a video about quantum physics. It should be the beginning of any research methods class.

Thanks to Twitter user @c4chaos for pointing to a link that lead me to this video.

“Utter bullocks”

I’m amused by this expression used in a comment on a RWW post titled: Study: 93% of Americans Want Companies to Have Presence on Social Media Sites.

I also believe it’s the perfect response to the report of this study, as presented in the RWW post. I don’t know, the study might be brilliant. But that’s the problem, they don’t provide enough information so I can decide if it’s brilliant or not.

Two issues here:

1) understand the data before you make decisions based on it

2) even if the data is good & valid, don’t jump in and make decisions based only on statistics & demographics

1) understand the data before you make decisions based on it

Some questions to ask about these particular results:

• who are the 93% of Americans? There aren’t that many Americans online in the first place!!! (P.S.: cultural sensitivity issue: “America” includes Canda, and South American countries. Do you mean U.S. residents?)
• they probably mean 93% of survey respondents, I guess (guessing = bad sign in research)
• who are the survey respondents? Provide information about the sample:
  • How many people took the survey?
  • Who were these people? – provide demographics of the sample
  • How were these people recruited? (random sampling, stratified random sampling, convenience sampling, self-selection?)
  • Is the sample representative of the U.S. (or the Americas?)

These are just a few things I’d like to know before I’d spend a dime on a “social media presence”. And, as RWW writer Frederic L. points out, which social media sites? Twitter and Facebook are so different they might as well be two foreign countries!

2) even if the data is good & valid, don’t jump in and make decisions based only on statistics & demographics

My social media mantra is: It’s not about technology, it’s about culture.

Culture (social norms, etiquette, communication practices) emerges quickly around a social medium, and is specific not only to that medium, but also to sub-groups of users. So you can assume there are hundreds if not thousands sub-cultures on Facebook alone (about 100 million users worldwide).

An example: Befriending someone you haven’t met before is perfectly acceptable on Twitter, but creepy on Facebook.

So think about social media as a continent with many different countries and cultures. If you were to go to Romania (my native country), would you start doing PR & marketing armed with just some demographics produced by a poorly designed research study? I certainly hope not! I hope you’d take some time (a couple of years, say) to begin to get a grasp of Romanian culture before you dive in.

Same goes with social media. Start with your surveys, and make sure you understand what a good survey is. But do some ethnographic research, too (focus groups will do) before you spend that dime on your “social media presence.”

P.S.

Since I’m ranting, let me point out that the phrase “social media presence” is also … (see post’s title). It’s not about presence, it’s about engagement & conversation.