This methodology explainer is particular to the Canadian study. To learn more about the scope and theoretical framework of the JRP study in its totality, you can access information on the international website.
Sampling
There are twelve sites of study for the Canadian JRP project:
Television:
CTV: CTV News is the journalistic division of the CTV Television Network, Canada’s largest privately owned television network. The terrestrial network is a division of Bell Media, a subsidiary of publicly-traded BCE Inc., and offers programming in English. CTV’s national news subdivision offers viewers a nightly newscast (CTV National News) and is the number-one rated television newscast in Canada.
CBC: The Canadian Broadcasting Corporation is a government-funded public broadcaster and the oldest existing broadcasting network in the nation. The National is the flagship nightly newscast of CBC television, broadcasting six days a week from Toronto, and is a site of study for the JRP Canada project.
Global: Functioning as the news division of the Global Television Network, Global is a news and current affairs series with a national scope. Global News is owned by Corus Entertainment. The third site to offer a national perspective in Canadian television journalism, Global National is one of the top-rated news shows in the nation and airs daily.
TVA: TVA (short for Téléviseurs associés) is a franco-Canadian television network, owned by Groupe TVA, a publicly traded subsidiary of Quebecor Media. The regional network exclusively holds stations in Quebec, but has an Ottawa bureau, and produces the top-rated news show in the province, TVA Nouvelles (evening network edition), which is a site of study for JRP Canada.
Radio:
CBC: CBC Radio is the English-language division of the Canadian Broadcasting Corporation. Three English language networks exist under the CBC Radio umbrella including CBC Radio 1, CBC Radio 3 and CBC Music. Radio One is primarily responsible for broadcasting news and information to communities across Canada, and it is the home of the World at Six, airing Monday to Friday, a site of study for this project.
Radio-Canada: Ici Radio-Canada Première is a Francophone radio network and a division of Société Radio-Canada, the French-language national public broadcaster. It is often the only radio available in smaller French-language communities, reaching approximately 90 per cent of Francophones in Canada, and the largest French-language radio news service in Canada. The radio network’s flagship program L’heure du monde (a 60-minute news and discourse program that acts as an English counterpart to CBC’s The World at Six) is a JRP site of study. L’heure du monde airs Monday to Friday.
Newspapers:
Globe and Mail: Privately owned by the Woodbridge Company, the Globe and Mail is a Canadian newspaper and one of two national, daily newspapers in Canada with a Monday to Saturday circulation. Founded in 1844, the publication offers audiences international content and harbours a large staff of foregin correspondents with bureaus in Beijing, Washington, Johannesburg, London, San Jose and Rome.
National Post: The National Post is a broadsheet daily newspaper and one of the leading publications of Postmedia Network Incorporated. The paper was founded in 1998 and publishes Tuesday to Saturday, making it one of two national newspapers in Canada.The National Post was built around the Financial Post, which now serves as the paper’s business section.
Toronto Star: The Toronto Star is a local broadsheet daily newspaper with the highest weekly circulation in Canada and an international reputation and focus. While local coverage is Toronto-centric, the newspaper also has reporters in Vancouver, Calgary, Edmonton, Winnipeg, Ottawa and Halifax. The Toronto Star was founded in 1892. Torstar, the Canadian publishing company that runs newspapers and websites, was sold to investment company NordStar Capital in 2020.
Online Media
La Presse: La Presse + is a Montreal-based, French, digital news organization with a popular app and website that publishes daily news. Formerly a legacy newspaper, it ceased printing on weekdays in 2015 and on Saturdays in 2017. La Presse was previously owned by the holding company Power Corp. of Canada but is now under the ownership of a non-profit social trust. Stories coded for the JRP project were captured on lapresse.ca.
CBC.ca: CBC.ca is the exclusively online Anglophone iteration of the Canadian Broadcasting Corporation. Launched in 1996, the online web news service is one of the nation’s most visited websites. It is the most used news site in Canada with a national focus. Through the webpage, users have access to top national and localized municipal news.
HuffPost Canada: HuffPost Canada was one of the most popular new sites in Canada and the first international edition of Huffington Post, an American news website that produces and releases exclusively online news content. It was launched in 2011 as a Canada-centric incarnation of the HuffPost. Following Buzzfeed’s acquisition of HuffPost Canada from Verizon in 2021, HuffPost was shuttered.
Data Collection
There were 14 dates of data capturing throughout 2020 (two constructed weeks), with at least one data collection date in each month (January 29; February 20; March 20; April 11; May 5; June 8; June 30; July 7; August 5; August 27; September 25 ; October 17; November 15; December 14). Content capturing for sites of study varied based on platform of delivery. For online sites, research assistants created pdfs of homepages and then captured pdfs of all of the stories on those homepages starting at 11 a.m. and 11 p.m. on data collection days. For radio and television broadcasts, automated recordings were set up at Toronto Metropolitan University. Newspapers being studied were collected by researchers on the days of publication. In some instances, when researchers were unavailable to get printed copies, photocopies or printed Press Reader copies of newspapers were used.
Only stories that were created by the organization itself (in whole or in part) were included in the analysis. For example, stories with a CP reporter byline or those taken from another media outlet or news agency that were published by the unit of study were not coded; however, stories “with files” from CP or another agency but with a reporter byline from the outlet of study were coded. All types of news stories were included in the sample, from politics to sports to lifestyle to business; however, no opinion pieces or editorials were included in the sample. For broadcast outlets, introductions to stories and any back and forth between a reporter and anchor were together included as one story. Live hits from the scene of a story that only included a stand-up or on-site interview were also coded as a story. On-set interviews between an anchor and guest were also coded.
Early in the capture, it was noted that some stories on the HuffPost homepage were significantly dated; initially, we assumed this was because the stories were still attracting traffic and it was an editorial choice to keep them on the homepage. However, participant observation and interviews revealed that this was actually just a matter of the editor not having time to physically remove the stories from the homepage/forgetting to remove them. In order to manage the number of stories that needed to be coded and to ensure data being analysed would reflect the news-day being studied, stories that originated more than five days from the capture date but were still on a news site’s homepage were eliminated from the sample for all sites of study. For example, if the capture date was the 29th, only stories dated from the 25th-29th posted on the homepage were coded. In total, 3,727 stories from the four platforms were analysed by the Canadian team.
Analysis
The vast majority of content analysis using the codebook started in September 2020 and was not completed until October 2021. Content analysis was a complex and time consuming process. Masters and undergraduate students were hired as RAs, with undergrads primarily focusing on capturing/recording content during collection and Masters students coding, with the exception of one undergrad-student coder. In order to ensure consistency of coding, ongoing training sessions were held, intercoder tests run periodically, and a shared Google doc created to address coding questions as they happened/ensure all coders could address any issues/were coding variables in the same manner. The PI for Canada made the final call on questions concerning coding and would consult with the project lead, professor Claudia Mellado of Pontificia Universidad Católica de Valparaíso, Chile, when issues arose that were not easily resolved via the codebook. There were both French and English coders due to Canada’s media system which has both French and English journalistic outlets.
The medium frequency distribution for stories coded is as follows (story platforms are being weighted in much of the analysis):
Table 1. Role Performance Indicators
| Role | Indicator |
| Interventionist | Journalist’s point of view Interpretation Call to action Qualifying adjectives First person |
| Watchdog | Information on judicial/administrative processes Questioning by the journalist Questioning by others Criticism offered by the journalist Criticism offered by others Uncovering performed by the journalist Uncovering performed by others Reporting on external investigation Investigative reporting |
| Loyal-Facilitator | Defense/support activities Defense/support policies Positive image of the elite Progress/success Comparison to other countries National triumphs Promotion of the country Patriotism |
| Service | Impact on everyday life Tips and advice (grievances) Tips and advice (individual risks) Consumer information Consumer advice |
| Infotainment | Personalization Private life Sensationalism Emotions Morbidity |
| Civic | Citizen reactions Citizen demand Credibility of citizens Education on duties and rights Local impact social community impact Citizen questions Information on citizen activities Support of citizen movements |
Survey Distribution
The Canadian survey was housed on Opinio, an internal platform at Toronto Metropolitan University. There were two versions, one English and one translated from English to French. Following a review of the data-gathering methodology by TMU’s Research Ethics Board, invitations to participate in the survey were emailed to journalists identified by their bylines in captured text-content or sign-offs or name-keys in captured broadcasts. The list also included employees identified on news organization websites or social media feeds. Surveys were sent out between September 2020 and June of 2021. The timeframe of survey distribution was extended in order to attain the participation of the required number of journalists.
There was resistance from journalists with regards to taking part in the survey. Several who emailed/spoke with researchers expressed concern that there was no room to provide context for answers, that the questions were not relevant to their work, and/or that the questions were framed in a way that supported the value of traditional standards such as objectivity that are increasingly considered problematic amongst Canadian journalists and journalism educators. The researchers were able to explain the relevance of the survey and/or address the issues of some journalists who had initial reservations with regards to completing the survey (although no identifying information was tracked within the survey so researchers cannot confirm who did or did not complete surveys). Other journalists refused to participate, but did give their time for in-depth interviews. One media organization discouraged its employees from completing the survey due to concerns confidential elements of their practice would be shared, but later agreed to allow, but not encourage, employee participation.
As not all journalists who started but did not complete the survey contacted researchers, it is impossible to know which of the identified issues, or other unidentified issues, prevented survey completion; however, the length of the survey itself may have proved problematic. Of the 243 journalists who started the English survey, only 113 finished. For the French survey, there were greater challenges: only 13 journalists started the survey, with only five completing it (one from Radio Canada completing the English survey). In order to attain the designated number of required responses for each site of study, the research team submitted an addendum to its research plan to the Research Ethics Board at TMU in order to gain approval to phone journalists to recruit survey participants. The minimum number of surveys needed from each site of study in order for a site of study’s data to be included in the international dataset was based on newsroom size as defined by the international team: 50 journalists or smaller for a small newsroom (minimum four survey responses); between 50 and 200 journalists for a medium-sized newsroom (minimum eight survey responses); and more than 200 journalists for a large-sized newsroom (minimum 12 survey responses).
| Media outlet | Actual # of surveys completed |
|---|---|
| CTV – The National | 10 |
| CBC TV – The National | 9 |
| Global National | 11 |
| TVA Nouvelles – 6 p.m. newscast | 6 |
| CBC Radio – World Report | 5 |
| Radio-Canada – L’heure du monde | 7 |
| Globe and Mail | 21 |
| Toronto Star | 32 |
| National Post | 6 |
| LaPresse | 8 |
| cbc.ca | 12 |
| HuffPost Canada | 6 |
| Total | 133 |
CBC was a particular challenge in terms of determining the required number of surveys as we were unable to confirm exact numbers of journalists versus technical staff working in the national newsroom and there were three different CBC sites of study (one radio, one television, one online). Three sites of study were chosen within the organization due to its significant impact on Canadian media and its role as a public broadcaster covering regions/stories in Canada that are not always examined by other news outlets. It airs the only national daily-news radio program and its news website is the most visited in Canada. Potential survey respondents noted in emails that platform designation was no longer used at the organization. Therefore, CBC on the whole was considered a large newsroom.
However, as the international study had a 12 survey response minimum for any organization deemed large, and each CBC platform/unit of study was considered to be part of a “large” organization due to the framework of the international study, only the survey responses for those who selected cbc.ca as their primary mechanism of delivery were included in the international data, because only that platform met the minimum number of survey respondents for a large organization. For national-level analysis, however, as there are a sufficient number of responses coming from each site of study (platform) to offer a fair representation of practice for each medium being studied, the Canadian team will be using a dataset that includes all of the CBC responses in some instances.
For The National Post, we were only able to garner six survey responses. This was not enough for the minimum of a medium-sized newsroom to be included in the international data set. However, for analysis at a national level, these responses will be included in some instances.
Based on information collected on the Radio Canada website, the radio show L’heure du monde has a regular team of 11 journalists and a few dozen more occasional collaborators; therefore, even though Radio Canada would likely be considered a medium-sized newsroom, the seven survey responses from specifically targeted journalists is believed to be a fair representation of practice for this particular unit of study and data was included in both the international and national analysis.
Table 2. Survey Items on Role Conception and Perceived Enactment
| Roles | Role conception/perceived role enactment Items (survey) |
| Interventionist | Providing your opinion on the issues you are reporting, as an expression of approval or disapproval Providing your own proposals or ideas regarding potential policies or solutions to problems Calling the public to behave in a certain way regarding an event or issue Using evaluative terms about events or persons in your news reporting that reveal your own way of seeing things Campaigning for a political or social cause |
| Watchdog | Questioning the truthfulness of what powerful individuals or groups in society such as politicians and political parties, governing bodies, business, the church, and the cultural elite, say or do Uncovering illegal /irregular behaviour of powerful individuals or groups in society, such as politicians and political parties, governing bodies, business, the church, and the cultural elite Including information on judicial or administrative processes regarding powerful individuals or groups in society Providing information on abuses of power or wrongdoing based on your own extensive inquiry and research.Quoting sources that question, criticize or uncover wrongdoings of powerful groups or individuals in society |
| Loyal-Facilitator | Expressing positive feelings about being a citizen in your country. Presenting in a positive light public policies or official activities of governing elites. Favourably stressing the leadership, management skills or personal characteristics of institutional powers, economic leaders, and powerful elites. Looking for opportunities to share social, sporting, political, or economic achievements of your country. Praising the achievement of individuals, groups or organizations in your locality or country. Defending your country’s national values |
| Service | Indicating how a particular event or action might influence the day-to-day lives of ordinary people. Giving the public tips and practical guidance on how to deal with everyday life problems. Providing consumer information about trends, advances and new products or services. Appealing to the audience to help other people with specific and personal problems they may be facing. Helping the audience to recognize the good and bad quality of specific products or services |
| Infotainment | Featuring individuals in the news, by providing information on their intellectual, physical, or social characteristics. Including explicit references to emotions of people. Including information on the private life of people being covered in the news. Including details to heighten the impact of stories about violence, crime, extreme poverty, sex, etc. Using humour, exaggeration, sarcasm, or other storytelling devices to provide information in an entertaining way |
| Civic | Informing people on their economic, social and/or political duties and rights.Including background information and in-depth context for citizens, regarding specific events in the news Mentioning the impact of decisions on the periphery of your country beyond the capital city and other main cities Including different perspectives in the news in order to better inform citizens. Depicting the impact of authorities’ decisions on specific socially defined communities such as social classes, sexualities, races or ethnicities Informing the public about citizens activities Explicitly supporting citizen movements and demands |
Multi-level/nested analysis with mixed methods
Data gathered through the content analysis was used to establish journalists’ performance; survey data was used to establish both an individual journalist’s conception of their roles and enactment of journalistic roles at an organizational level; a potential gap in performance versus conception was measured by comparing content analysis (what journalists did) to survey responses (their perception of what journalistic roles were important). Socio-cultural data from a variety of sources, such as the RSF World Press Freedom index, were also used to create variables for each country to allow for more contextualized analysis of findings. All international teams have access to shared, cleaned, and merged data collected by the 37 countries involved in the project in order to perform cross-cultural analysis.
The Canadian study included REB approval for both participant observation and semi-structured interview components, with the hope a mixed methods approach might bring greater insight into data gleaned from both the content analysis and surveys. However, only one organization, HuffPost, agreed to allow us into the newsroom to observe practice, and the pandemic limited study there to one day that included observation of the morning news editorial meeting, shadowing editors and reporters, and informal interviews with six journalists (not including additional meetings beforehand aimed at gaining participation).
Additionally, there were semi-structured interviews with nine journalists who created content for a variety of platforms/sites of study for this project. Seven interviews were arranged after potential participants contacted the researchers with questions about the survey or concerns about the nature of the survey; two interviews were follow-ups to participant observation.