Musings on Critical Realism

For the past two weeks, I have been reading about research philosophies and methodologies to help design my research.  It is a bit difficult, given that I am working with an interdisciplinary topic that crosses multiple areas. My topic is on the 'usability of disaster mobile applications from the perspective of the citizens'.

There are two strong research paradigms: positivism or interpretivism. I was on the fence on which one I will adhere to... Because, even just on the top level, I can cut my topic into three broad themes: [1]  mobile applications (technology), disaster management  (social science/ management), [3] citizens' use (computer-human interaction). So there is the technical, social, AND the interaction between the two. So... am I primarily dealing with information systems which has a strong predominance on positivism? Or am I dealing with the social concept of technology use which might lean more towards an interpretivist stance? 

However, after further reading, I realize that these are more than two options. There are also other concepts such as pragmatism, critical realism, and others. So far Critical Realism (CR) seems to suit my research world view.

Critical realism takes on the realist stance -- where reality is believed to exist outside human perception (hence 'realism'). But knowledge of this reality can only be perceived through human experiences where knowledge is always local and historical; viewpoints (scientific, political, social, etc...) are all valid but to some extent fallible (hence 'critical'). Critical realism is "ontologically bold but epistemologically cautious." (Wynn and Williams, 2012). 

How reality is viewed by critical realism:

CR acknowledges that there is an intransitive domain where there are enduring properties. In the realm of the real, there are inherent powers and tendencies of things. In the realm of the actual, these tendencies are triggered into events (whether observed by us or not). The empirical is the realm where we acquire our knowledge, where we get to measure and to perceive the experienced events. 

Despite a realism ontology, CR allows for epistemological relativity (Zachariadis, Scott & Barret, 2013).  The process of our scientific knowledge is in the transitive domain where it is "historically emergent, political and imperfect"(Zachariadis, Scott & Barret, 2013). With a relative epistemological stance, CR supports mixed methods research. CR acknowledges that in order to access different epistemological characteristics of different type of objects we can use a range of research methodologies.  (Mingers, Mutch & Willocks, 2013). 

Some key points to consider further

• retroduction
• causal tendencies

Important Readings (some read and some still to read)

• Critical realism and information systems

Carlsson, S. a. (2011). Critical Realist Information Systems Research in Action. In M. Chiasson, O. Henfridsson, H. Karsten, & J. I. DeGross (Eds.), Researching the Future in Information Systems (pp. 269–284). Turku: Springer. http://doi.org/10.1007/978-3-642-21364-9_17

Dobson, P. J. (2012). Critical realism and IS research: Some methodological implications. In M. Mora, O. Gelman, A. Steenkamp, & M. Raisinghani (Eds.), Research Methodologies, Innovations and Philosophies in Software Systems Engineering and Information Systems (pp. 63–81). Hershey, PA. http://doi.org/10.4018/978-1-4666-0179-6.ch004

Mingers, J., Mutch, A., & Willcocks, L. (2013). Critical realism in information systems research. MIS Quarterly, 37(3), 795–802. 

Smith, M. L. (2006). Overcoming theory-practice inconsistencies: Critical realism and information systems research. Information and Organization, 16(3), 191–211. http://doi.org/10.1016/j.infoandorg.2005.10.003

Wynn, D. J., & Williams, C. K. (2012). Principles for conducting critical realist case study research in information systems. MIS Quarterly, 36(3), 787–810.

Zachariadis, M., Scott, S., & Barrett, M. (2013). Methodological implications of critical realism for mixed-method research. Management Information Systems Quarterly, 37(3), 855–879.

• Design science

Carlsson, S. A. (2010). Design science research in information systems: A critical realist approach. In A. R. Hevner & S. Chatterjee (Eds.), Design research in information systems: integrated series in information systens (Vol. 22, pp. 209 – 233). http://doi.org/10.1007/978-1-4419-5653-8

Hevner, A. R., March, S. T., Park, J., & Ram, S. (2004). Design science in information systems research. MIS Quarterly, 28(1), 75–105. http://doi.org/10.1007/978-1-4419-5653-8_2

Peffers, K., Tuunanen, T., Rothenberger, M. A., & Chatterjee, S. (2007). A Design Science Research Methodology for Information Systems Research. Journal of Management Information Systems, 24(3), 45–78. http://doi.org/10.2307/40398896

• Mixed Methods

Venkatesh, V., Brown, S. A., & Bala, H. (2013). Bridging the qualitative-quantitative divide: guidelines for conducting mixed methods research in information systems. MIS Quarterly, 37(3), 855–879.

• 
  

Progress: 18 July 2012

I'm getting some progress with my work and just happy to present these diagrams that seem to make the sense out of my work (more than my write up). After much deliberation on my research structure, I have decided this will by my methodology scheme:

Click to enlarge

Also I've finally made sense of the documents I'm using! I've tied it up with my discussion on UNDP. Basically it's going to be the documents on the programme level + UNDAF. Still trying to decide whether to include documents on the project level.

Click to enlarge

Research Structure

This picture from Krippendorff shows an idea of how I am going about my research method:

Click to enlarge

Although, this doesn't entirely capture what I want to do... Since I'm not sure I'm using content analysis alone. I think it is more like I'm building cases out of the documents; and I'm using content analysis + (something else, i.e. discourse analysis/semiotics/observations?) to make sense of the documents...

So the picture below maybe a better representation (my inputs in red)...

Click to enlarge

I plan to construct 15 mini (pseudo) cases to represent the risk management and outcome phenomena in each country; and I would then compare those similar phenomena to make an inference.

Does that sound sensible?

---

KRIPPENDORFF, K. 2004. Content analysis : an introduction to its methodology, Thousand Oaks, Calif., Sage.

Documents Thus Far

I'm streamlining my data... So far I have shortlisted my sources of documents to the following categoreis

Region: Europe and CIS
Timeframe: Programme cycles that fall between 2005 to 2011
Countries: Albania, Armenia, Azerbaijan, Bosnia and Herzegovina, Georgia,  Kazakhstan, Kyrgyzstan, Moldova, Romania, Serbia, Tajikistan, Turkmenistan, Ukraine, Uzbekistan and the former Yugoslav Republic of Macedonia.

Documents to be analysed:

1. UNDAF
2. CPD (past cycle and on-going cycle)
3. CPAP (past cycle and on-going cycle)
4. ADR (if any) - or - Country Programme Performance Summary (if without ADR)

Given the above list... I am looking potentially at 90 document (1 UNDAF, 2 CPDs, 2 CPAPS, 1 ADR/Summary for each country) already... I'm still deliberating if it will be necessary to look into the additional documents. Aside from the constraint of time and the sheer amount of volume of documents, I've listed my reservations about some of the documents.

1. Outcome Evaluations - Reservation(s): Not all country programmes have outcome evaluations for the specified period. And only selected outcomes are evaluated. Advantage(s): it would list the projects and outputs that contribute to achieving the outcomes.    
2. Project documents - Reservations(s): I'm still undecided whether I should look in depth into project documents since I'm looking at the programme level; maybe I could use it as a support. There is a risk that I would look at the thesis on the project level. Advantage(s): since the programme documents (so far) have not given me quality data for risk management, maybe the project documents could give me insights:  
3. Project evaluations - my reservations and advantages are the same as #2

I guess the question here really is:

Is the country programme documents sufficient to prove a point? If not then I would need to look into project documents.

---
Here are the sources for the documents

UNDP Europe and CIS Country Programme documents (UNDAF and CPD):
http://europeandcis.undp.org/news/show/8EEC429C-F203-1EE9-BB5E744EAA529CF8

ERC Website (UNDAF, Outcome and Project Reports):
http://erc.undp.org/

Assessment of Development Results Reports
http://web.undp.org/evaluation/country-evaluation.html

Also know that... not all country offices would have ADRs, but instead would have a Country Programme Performance Summary. I haven't found a repository... but through persistent search using the keywords "UNDP" and "Country Programme Performance" all the relevant summaries can be found.

Drafting my Research Design

My data will come from official UN documents. Through the months, I have been mapping the relevant documents and seen some samples of them. Before I officially embark on my formal data collection... I had to be more clear about my research method. After a week of extensive reading I think I have come to understand my research structure a bit better. Here is the line of thought of my research:

I also found a Krippendorff's book on content analysis really helpful. It gives me the academic/theoretical support of what I want to do. Here is a conceptual process of my research methodology:

Krippendorff identifies that in content analysis, there are three types of units: 

1. Sampling unit -  units of selection; what documents are going to be sampled and analysed? (i.e. CPD, evaluation report)
2. Coding unit - units of description; what is going to be coded/measured/analysed? (i.e. textual presence of risk management, mentions of programme success)
3. Context unit - units that delineate the scope for coders in recording the coding units; what context is the coding unit coded? (i.e. the logframe is a context unit, textual presence of risk management is the coding unit)

The examples above are working definitions of my units of analysis, I still have to work on clarifying them.

As for my sampling, it is clear that I'm using relevance sampling:

"Relevance sampling is non-probabilistic. In using this form of sampling, an analyst proceeds by following a conceptual hierarchy, systematically lowering the number of units that need to be considered for an analysis. The resulting units of text are not meant to be representative of a population of texts; rather, they are the population of relevant texts." (Krippendorff, 2004)

----
References:

GRAY, D. E. 2009. Doing research in the real world, London, SAGE.

KRIPPENDORFF, K. 2004. Content analysis : an introduction to its methodology, Thousand Oaks, Calif., Sage.

KRIPPENDORFF, K. & BOCK, M. A. 2009. The content analysis reader, Thousand Oaks, Calif., Sage Publications.

Finding my Unit of Analysis

For the past few days I have been trying to define my case. Alec has blogged about Yin's illustration of different types of case studies. One thing for sure, I am doing a multiple case study. But I'm just having harder time to frame up what my unit of analysis is... what is exactly the case I am studying?

Is it the country programme risk management process? and the country project risk management process as an embedded unit of analysis?

or 

Is it the UNDP risk management process? and the country programme risk management process as embedded unit of analysis?

or 

Is it the project and programme risk management process congruency?

So confused!

Tried to write it up on the board... and I think I may have something here... somewhere.

I'm merely posting this so I won't lose the images...  I will stare at this for a while and maybe a thought might just jump in.

Ideas anyone?

What is Grounded Theory?

It is not a theory! Rather, it is an approach to the generation of theory out of data.

'Grounded theory has been defined as "theory that was derived from data, systematically gathered and analysed through the research process". In this method, data collection, analysis, and eventual theory stand in close relationship to one another'. (Bryman, 2012)

Source: BRYMAN, A. 2012. Social research methods, Oxford, Oxford University Press.

Using Documents as Data

I'm happy with my choice to do use archival documents as my primary data. Makes me feel a bit different :) Everyone else seems to be opting for using surveys. But it is not an easy choice. Reading into some literature, I have a lot of work to do... and so many decisions to make for my methodology!

Here are some concerns that I need to look at when using documents as data.

1. Level of Analysis - Few or ,many (or both)?
• Few - information from a single or few organisations (concentrate on a few country offices)
• Many - information from many organisations (a sample of country offices in a region)
2. Input Method - Read or measure (or both)?
• Read - using large amounts of archival information, make discoveries through managed pattern of reading
• Measure - Use social scientific 'coding'; insight comes from attention to patterns
3. Causality Theory - Descending or ascending (or both)?
• Descending - Macro patterns are expected to explain micro processes
• Ascending - Micro processes develop on their own, but are incorporated at a higher level of social organisation
4. Measurement Theory - Objects or relations (or both)?
• Objects - analysis focuses on connecting attributes to outcomes
• Relations - relations (rather than characteristics) of objects are expected to yield explanatory value.

Also reading about archival analysis raised a lot more questions. Just clear that there is a lot that I don't know. I read a lot of analytic methodology terms that I do not quite understand, but I may use it in the future:

Content analysis

Semantic grammars

Semiotics

Multidimensional scaling

Sequence analysis

Boolean algebra

Galois lattice

Correspondence analysis

Hierarchical classification model

---

Source:
VENTRESCA, M. J. & MOHR, J. W. 2002. Archival research methods. Blackwell companion to organizations, 805-828.

03 April 2012: ReMe

Today's ReMe was definitely better than last December's. The discussions are a bit more generic, but this time it got me to do some thinking about my project. 

Here are the highlights for the day:

• Quality of Data Lecture: its about using statistical techniques to data. Even though my data might be more qualitative than quantitative, I am sure that I will use some statistical analysis to some extent in my dissertation. It was a good reminder to me that aside from reading literature, thinking/searching of the possible data source, reading about methodology, etc... I also have to revisit statistics for use in my analysis. But I think that's still a bit far off, first I need to concentrate on setting the base of my study.
• Qualitative Analysis Lecture: the lecturer touched on archival data. The main point was these documents (i.e. UNDP CPAP, AWP etc..) were not written for the purpose of my thesis. The challenge is coding these documents, finding patterns and relating it with the literature. I wish she discussed about it to some more detail, but she didn't. The focus of this lecture was more on ethics and interviews... which could still be helpful for me.
• 'How to avoid Failing' Lecture: First, on a higher note... I will remember the 'golden thread', a distinctive thesis seems like there is a 'golden thread' linking every piece of the thesis beautifully; it flows from start to finish.  But the rest of the lecture is spirit dampening... A lot of DON'Ts and DOs regarding the dissertation structure. The main point is: do not write/include anything that does not add value to the thesis. 

I really don't like the 'how not to fail' theme, it dampens my spirit. After the ReMe session, I felt stressed, anxious; and up to now a lingering feeling of silent panic. -.-

Keep Calm and Carry On...