Abstract

1 Introduction

2 Is an ethnographic study the right choice?

3 Planning an ethnographic study

4 Implementing your ethnographic study

5 From Data to Text

6 Ethnography and Research Ethics

7 Final comments, Further reading and References

5 From Data to Text

The first part of the term Ethnography (“ethnos”) means culture, while the second part (“graphy”) refers to the writing, the presentation of information about the culture or practices observed. When using ethnography as a research method in software engineering, the presentation of the results is in the form of articles, and less often in the form of a monograph. The challenge often is to reduce the rich experiences from the field into what can be presented in the format of a conference or journal submission. The more organised the ethnographer, the easier the task of analysing the collected data [21]. We first present the analysis process and thereafter discuss how to relate it to the software engineering discourse.

5.1 Reflective and inductive analysis

The analysis of ethnographic data is a reflective and inductive iterative process that starts in parallel to field work. It is possible and even recommended that ethnographers write temporary notes (often called memos) about what is observed in the field. The information captured in memos will be used backwards to identify where else a similar pattern showed up in the field work so far, and forwards to inform the following period of data collection. For instance, as mentioned in the previous section, de Souza and Redmiles [57] interviewed experienced developers to find out whether their experience was similar to novice developers.

These memos are also an important starting point for a more formal and systematic analysis. But where to start the analysis is a conundrum when faced with so much rich data. One useful technique is to look for “rich points” [2]. These are surprises or notable insights that researchers experience while in the field. Often they may manifest as initial impressions when entering a new event or context (although any such impressions should be systematically validated). These may lead to narrative accounts of everyday practices, or to identifying patterns that appear across situations. The former is not simply a description of what was observed, but includes a synthesis and interpretation of the data. Being careful with language both during analysis and in the ethnographic account is very helpful here: “certain birds can solve problems better than some dogs” carries a different meaning to “birds are smarter than dogs”. In the process of capturing and reporting findings, it can be easy to summarise or abstract something that accidentally leaves out important context or nuance.

Ethnography, like other qualitative research methods, often employs coding techniques to identify patterns across situations, i.e., data are micro-analyzed (line-byline) to identify codes. When coding, an ethnographer is actually making sense of all the data collected. These codes are put together under a more abstract, high-order concept to explain what is going on [8][30]. For instance, [57] de Souza and Redmiles [57] created a code to indicate the concept of “back merges”: the fact that a software developer had to incorporate code changes from other developers into their own workspace before integrating their code into the main codebase. Codes are created to minimise the number of elements that the researcher needs to consider. After creating codes, and often in parallel to them, codes can be grouped together to suggest a category, an even high-order concept. In this case, de Souza and Redmiles [57] created the category backward impact management to indicate how a software developer continuously assesses the impact of the work performed by other developers on their own work, and the appropriate actions developers adopted to avoid such impact. In other words, a “back merge” (code) is a form of backward impact management (category).

However, coding is not necessarily only informed by the data. For example inductive codes might be combined with codes that relate observations to theoretical concepts. Likewise, there is no requirement for code coverage, as the data collection especially in the introduction of the observation is not yet focussed on what emerges to be the main theme. For a more detailed coverage of qualitative analysis see the chapter by Treude in this volume and the Further Readings provided in section 7. In general, the goal of the analysis is to invoke all data collected (field notes, documents, code fragments, informal and formal interviews, etc) to draw a clear picture of how the field site works, i.e., why the software developers do what they do. When envisioning this big picture the ethnographer’s attention will often wander through the data and will make logical leaps that lead to interesting insights. However, they must also “backtrack to see whether the data will support these new ideas or invalidate them” [20]. Ethnographers and qualitative social scientists talk about a ‘dialogue with the data’. During analysis the researcher moves between formulating reflections and interpretations leading to codes, which then are tested with the empir-ical data [42]. This also includes data triangulation, i.e., exploring different sources of data and making sure they support or at least don’t refute the result. For instance, the results from observation can be combined with the analysis of interviews, or documents. In addition, an experienced ethnographer will use “member checking”, i.e., will share their writing, memos, or any form of temporary results with trusted informants to gather feedback and find out whether their understanding of what is going on in the field site is recognised by the informants. It is important that the description of what was said and seen in the site be recognized by the informants, even if they do not fully agree with the interpretation of these results

One way data analysis is taught in social sciences is to analyse data together. Here researchers take an anonymised version of their own field material, an interview transcript, an instant messaging protocol, or a video or audio recording and analyse it with others. Of course it would not be possible to reconstruct the full ethnographic experience. However, in joint analysis, the teacher could point to relevant cues that connect the specific part of the field material to the broader context of the empirical work and that way illustrate how the brought picture gained through ethnographic field work allows them to then make sense of the specific document or conversation.

5.2 Writing Ethnography for Software Engineering Audiences - Reporting the Results

The results of analysis need to be related to software engineering literature to argue how the findings lead to new insights for the software engineering community. This has traditionally been difficult to achieve, as many software engineering researchers saw it as the goal of software engineeringto improve industrial software development practice rather than to understand why practitioners do things the way they do. In recent years, though, this has started to change. Ethnography and qualitative empirical research are more accepted as a means to understand development practices better, and hence to improve understanding which can then inform the development of new tools and techniques. Although the overall theme of the research is likely to have been established before field work starts (if only to inform the choice of field site), the specific interesting aspects that support new insights for the software engineering community will evolve during field work and analysis. Because of this, the researcher might have to complement the usual initial literature study with a short field study to establish the value of their chosen research focus, especially for a Master’s or PhD project.

Sometimes, additional analysis or even additional research is necessary to estabish an argument for the relevance of the results. In this case, ethnography is used in combination with other studies using different techniques. An example of this is described in Plonka et al. [46]. In this research ethnographic field work was complemented by video recordings of pair programming sessions. The analysis of ethnographic data led to the identification of a number of ways in which knowledge transfer takes place during pair programming. This in turn was supported by an in depth interaction analysis of the video recordings, which both illustrated and deepened the understanding of knowledge transfer in pair programming and led to some practical recommendations.

An example where additional research developed into an empirical (sub)project in its own right is Unphon and Dittrich’s article [63]. They present an interview study that was based on a finding in the above-mentioned ethnographic-grounded action research study. Unphon termed the observation of the technical lead communicating the architecture through one on one discussions ‘walking architecture’. The interview study established that this practice was wide-spread in software product development organisations and also uncovered some of the rationale behind these practices. When writing up the study, the method section needs to state the basic information about the site and duration of the study. But also a detailed account is needed of the field work methods, the analysis process and the measures implemented to assure the trustworthiness of the qualitative research. Often reviewers will ask for numbers, e.g. hours/days of observation, number of documents analysed, and even the size of the system being developed by the team. These can be included to characterise the study itself, but do not be tempted to analyse these numbers quantitatively as part of the results. Also the measures implemented to assure trustworthiness (see subsection above [47]) need to be presented in detail and examples of how these methods were applied in this study will often be expected.

One dilemma is the conflict between the need to provide rich descriptions and the result orientation of the software engineering community. Passos et al. [41] also mention this as one of the challenges of ethnography in the context of software engineering. Rich descriptions allow the reader to follow the inductive reasoning that led to the findings that support the insights. Software engineering reviewers often find these rich descriptions too long and uninteresting. Here the qualitative researcher will have to strike a balance between the quality of the account and the requirements of the community. Using tables, charts or other techniques to summarise context and findings can help establish such a balance.

• Writing as a part of interpretation and reporting is not always comfortable for those with an SE background, so offer writing guidance. • Language is important in the data so include some analysis exercises that focus on the use of language.

• First impressions can overshadow later observations SO look for confirming AND disconfirming evidence. • Succumbing to confirmation bias SO seek confirming AND disconfirming evidence for your interpretations. • Getting hung up on too much detail SO sit back and consider the wider picture. • Making too much of the detail of everyday practice rather than why things are the way they are SO get into the habit of regularly asking yourself “why”.

Authors:

  1. Yvonne Dittrich
  2. Helen Sharp
  3. Cleidson de Souza

This paper is available on arxiv under CC by 4.0 Deed (Attribution 4.0 International) license.