Inside the Research Project Examining Hidden Bias in News Coverage

Table of Links

1. Abstract and Introduction
2. Related Work
3. Feedback Mechanisms
4. The NewsUnfold Platform
5. Results
6. Discussion
7. Conclusion
8. Acknowledgments and References

A. Feedback Mechanism Study Texts

B. Detailed UX Survey Results for NewsUnfold

C. Material Bias and Demographics of Feedback Mechanism Study

D. Additional Screenshots

A Feedback Mechanism Study Texts

Data Processing Agreement

Who are we and how do we use the data we collect from you through this survey? This research study is being con- ducted by the Media Bias Research Group. We are a group of researchers from various disciplines with the goal of developing systems and data sets to uncover media bias or unbalanced coverage in articles. This study is anonymous. That means that we will not record any information about you that could identify you personally or be associated with you. On the basis of the collected data, we aim to publish scientific papers on presentations of articles that help to detect biased language, but these publications do not allow any inference to you as an individual. Once the study is published, the anonymized data might be made available in a public data repository. Your rights to access, change, or move your information are limited insofar as the data may no longer be modified after the data has been published in anonymized form. The reason for this is that we need to manage your in- formation in specific ways for the research to be reliable and accurate. Once anonymized, we will not be able to delete your data. The study itself is not hosted on Prolific, but on a dedicated external server. Once the survey is complete, you will be shown a unique code that you can enter in the Prolific form. Participation in this study is voluntary. You may choose not to participate and you may withdraw at any time during the study without any penalty to you. If you have any questions about the study or study procedures, you may contact the Media Bias Research Group, [email protected]

•   I agree to the processing of my personal data in accordance with the information provided herein.(Checkbox)

Demographic Survey

1. What gender do you identify with? (Female, Male, Other, Prefer not to say)

2. What is your age? (Input field for number)

3. What is the highest level of education you have completed? (8th grade, Some high school, High school graduate, Vocational or technical school, Some college, Associate degree, Bachelor’s degree, Graduate work, Ph.D., I prefer not to say)

4. What is the level of your English proficiency? (Proficient, Independent, Basic)

5. Do you consider yourself to be liberal, conservative, or somewhere in between? Please slide to record your response. (Very liberal to Very conservative, -10 to 10 point slider)

6. How often on average do you check the news? (Never, Very rarely, Several times per month, Several times per week, Every day, Several times per day)

Info on Media Bias

Before you can start we will now provide you with a few examples that should help you to understand possible media bias instances better. For each example, a sentence with a biased word (blue colored) is shown first followed by its impartial representation (green colored). Please note that bias is different from negative sentiment. Bias is ambiguous and subtle, it can be positive, negative, or not even have a particular sentiment but it still can imply or intensify the opinion/emotion

Subjective Intensifiers:

Schnabel himself did the fantastic reproductions of Basquiat’s work.

Schnabel himself did the accurate reproductions of Basquiat’s work.

Strong labels:

’The people want the Truth!’: Trump gloats over the loss of American media jobs.

’The people want the Truth!’: Trump tweets over the loss of American media jobs.

One-sided terms:

Concerned Women for America’s major areas of political activity have consisted of opposition to gay causes, pro-life law...

Concerned Women for America’s major areas of political activity have consisted of opposition to gay causes, anti-abortion law...

Attention Check on Bias

How is bias connected to sentiment? Based on the information that was provided to you earlier, please select the correct option.

•   Bias is the same as negative sentiment.

•   Bias can be both positive, negative or even not have par- ticular sentiment. (correct answer)

•   Bias is the same as positive sentiment.

•   Bias is not connected to sentiment at all.

Trust Check

Can we trust your data for scientific research? For example, if you failed to pay attention to some questions, please answer ’No’. Please answer honestly, you will receive full payment regardless of your answer. Please select one option. (Yes, you can trust my data for scientific research. No, you may not want to trust my data for scientific research.)