Research in the Information Age

Lecture #6 Data Collection, Analysis and Representation

An on-line information source provided by Bryn Holmes and Damian Gordon

| Intro | Lit Survey | Lit Review | Methods | Data Collection | Findings | Discussion | Conclusion

 

The use of Information Technology can aid research by aiding in supporting lower-level skills and thus allow researchers to focus on higher-order thinking skills.


Information & Applications

The methods you use will make or break the robustness of your findings and although the wide variety of qualitative and quantitative methods used in education can be confusing they do allow you to chhose a method appropriate to your area.

Information

The use of Information Technology can aid research by aiding in data collection and storage and analysis of data.

This module is designed for students doing research in either computer science or education or the M.Sc. in IT in Education which combines the two fields. Although the two are considered quite different there are commonalities in data collection and in keeping with the underlying goal of the M.Sc., to create a synergy between the two fields, common research designs will be explored here to support cross-fertilization of ideas and practice.

A common model of research in computer science is that of project implementation. Usually a problem drives the work and the resulting program explores solutions. If possible, data may be collected on the project once it is completed. In education one might look at the general issues of teaching and learning, or classroom activity, as projects that have already been implemented. The underlying question for most of educational research is whether or not these projects are successful and to examine and evaluate the outcomes.

A new method of teaching maths, for example, might be implemented using a IT- rich environment. Educational resarchers could explore a wide variety of issues in such a project ranging from: national and school policies on mathematics, the evolution of the curriculum, the role of ICTs in instruction, teaching practice in a changing world, teacher-pupil interaction, student-student interactions and combine any of these topics with overall issues of concerning gender, disavantaged youth, and multinational classroom environments etc.

Increasing the role of information and communication technologies are moving social science fields towards those of computer science and science in general. One traditional research model has been to remove subjects from their environment and test them by focusing on one or two variables in isolation. Limitations in applying results to the real world have created a search for more tenable research paradigms that seak to undercover rather then eliminate underlying factors. Although there have always been experimental learning methods that have been designed and tested on new audiences it has been difficult for any researcher or research centre to implement any project on a wide enough scale to get significant results. New technologies on the otherhand are allowing for the creation of learning environments and their potential dissemination world wide.

Data Collection

The data you collect should lie at the heart of your research question. Successful data collection is a matter of making choices and understanding the relationship between data and findings. The more structured your experiment the easier data collection will be but the more risky it is that you may not get meaningful results. Less structured data collection on the other hand will yield rich data but analysis is much more difficult.

You can look at the same issuees in terms of how instrusive you plan to be i.e., how much you plan to alter the research environment or how much you hope to capture existing practice.

Data Collection

From Structured data collection (instrusive role of the researcher)

To Unstructured data collection (ideally invisable researcher)

Example

experimental design

control group and experimental group

semi experimental questionnaires

semi structured interviews, classroom observation schedule; analysis,

 

video taping an environment; participant observation, narrative analysis, discourse analysis
Types of results numerical findings numbers and opinions self-reported opinions

rich data on a huge number of cultural questions. Patterns of interaction, visual and auditory data etc.

 

Writing up

Experiment almost writes itself - equiptment subjects/ materials

 

a great deal of structure less structure - so harder to write up. most difficult to write up - options are nedless. 'Teasing out threads of interwined meaning.'
Key points

hard to desing easy to write up - can have no real findings

 

  most difficult to write up - options are nedless. 'Teasing out threads of interwined meaning.'

Data collection is essentially a balancing act. In the structured data collection all the hard work goes into the design of the structure.This style of data collection is closest to the scientific model. If carefully planned, and if luck holds, then the experiment will yeild stastically significant data. Writing up is relatively easy and flows from the structure of the documents. The tone tends to be objective and formal. The problems arise when there are no real results; when for example there was no difference between the control group and experimental group. Pilot testing is, therefore, essential for anyone contimplating a large-scale, highly-structured experiment. If you don't seem to be getting results from initial small-scale trials consider changing your topic or methods.

On the other end of the spectrum are the types of studies that have their roots in methods of anthropology and which tend to explore human beings in their nature environments. Here data collection, aided by advances in recording technology, can simply be a matter of setting up a small unobstrusive video camera or audio recorder. Videotaping human interaction even for an hour will yeild a myrad of topics to study. The difficulties lie in making sense of what has been recorded. Twenty interviews each lasting approximately one hour will yield approximately 400 pages of single-spaced transcribed data.

Triangulation of data

Triangulation of data refers to the use of a number of data collection instruments so as to get a multiple view point of the data itself. The instruments might include a case study, a questionnaire and interviews.

An example of two different types of data collection in a Meta

Teresa Logan-Phelan - Child & Adult Fluency-: The Child & Adult Fluency course is a third-level problem-based learning course for Speech & Language Therapy students, on clinical practice.

http://www.tcd.ie/People/Teresa.Logan-Phelan/VisionPage/Indexvision.html

Teresa carried out an interview and also explored

Appendix 1 Interview with lecturer

Q. Briefly describe the course in which you used the Web-based courseware?

“The fluency and fluency disorders course is designed to introduce students to the area of fluency as an attribute of verbal performance in communication, to become aware of the biological, linguistic, and environmental factors that are necessary for fluency, and to consider in depth the acquisition of fluency in language development. Fluency disorders encompasses four main areas: Developmental stuttering Acquired (adult) stuttering of neurological origin Adult-onset stuttering of idiopathic origin Cluttering Students have to acquire knowledge about the above areas in sufficient depth to allow them to differentiate and diagnose clinical presentations, to plan and undertake therapy strategies, and to evaluate progress. The prevention of the development of stuttering and the management of stuttering and communications are also part of the course.”

How did you integrate the Web-based course into the teaching of your course?

R. “In the first place, possibilities about developing Web-based courses were discussed with the departmental staff at Oulu University, some who had input into the Landscape of Future Education in Speech Communications Sciences I, II and III. When it was confirmed that students would have access to Information Technology facilities, course development was actively pursued, and students were informed about the course two weeks prior to course delivery. The students were asked to prepare by observing a set of videos on fluency and fluency disorders. The Web-based course was presented to students at the first lecture, and they were asked to complete preparation for the succeeding lectures.”

Q. What part did the Web-based course play in three modes of presentation normally used in the teaching of the subject? Did the web element reinforce, replace or enhance the following: Practical assignments Lecturers Tutorials

R. “In all three instances, the objectives of the web course would be to provide data in a format structured to facilitate student learning, to stimulate thought and reflection about the complexity of the theoretical issues to be applied to clinical practice, and to develop and reinforce important personal and clinical skills. Since the student contact time with the lecturer was limited to eight hours (instead of the usual twenty-four), elements of the web course can be interpreted as replacing direct teaching.”

Q. Was the Web-based course successful in presenting the course material in the three formats required ie. practicals, lectures, and tutorials?

R. "Yes, the design of the course was very successful in the presentation in all aspects of the course. The integration of the problem based learning method was well presented throughout the course."

Q. Please describe how you used the course in teaching?

R. “Reference was made to the course material, discussions were generated about the issues associated with assessment of fluency presentations in the video clips presented, video clips of Van Riper’s therapy was used to augment the lectures and courseware.”

....... Try and becareful of questions that can be answered with simple yes and nos. In Teresa's case she got valuable information from her subject 90% of the time, but if the person does not expand the entire interview may flow as follows.

Q. Were the learning objectives clearly defined?

R. “Yes”

Q. Do you feel there were enough tasks integrated into the course for the student to do?

R. “Yes”

Q. Do the tasks make the student think about the subject matter?

R. “Yes”

Q. Are the tasks relevant to the learning objectives?

.....

Appendix 2 Usability Questionnaire For Child & Adult Fluency Disorder Web Based Course Naturalness 1.

Is language used in course material

a. Appropriate Yes No

b. User Sensitive Yes No

c. Free of jargon Yes No

d. Free of abbreviations Yes No

2. Are font and style easy to read?

Yes No

3. Are headings & paragraphs easy to distinguish?

Yes No

4. Does instructional material move from left to right Allowing for easy eye movement?

Yes No

5. Is colour used appropriately ?

Yes No



Applications of this Information

Try to design a questionnaire or survey and see what type of information it gives you.


Search, Select & Judge

Look in ERIC for a study with the same methods as yours.

http://searcheric.org/


Explore, Test & Actively Engage with the Information

Ask MITE00s about their expeiences with designing their data collection.

Statistics

The following section on statistics - Damian has already covered this course last term

http://www.cs.tcd.ie/courses/mscitedu/2000/term2/l105/DataAnalysis/DataAnalysis.html

Quantitative Analysis

Data Analysis of qualitative data is very tricky.

Software packages (QUALPro, NU-Dist, etc.) can aid in analysis of interview data .Analysis of qualitative/narrative data can include coding/digitizing, a-priori-themes analysis, emerging themes analysis, pattern finding, and computerized data analysis.

ATLAS-ti,

NU*DIST,

The Ethnograph

HyperRESEARCH.

Or like I did, you can design your own analytical methods to suit your research question.

--------

 


Collaborate and Discuss

Recall - last week's groupd discussion about different research methods.


Analyse & Synthesize

Use the following checklist to reflect on your own data collection

Evaluation category

Notes

Were the M/T/IS employed to analyse the data clear?

General

What needs to be measured?

Which variables?

What scale?

What units?

Was it possible to measure anything at all?

General

When the data was being collected was it store in a manner that would allow for comprehensive analysis?

(for large data sets, stored electronically, etc.)

General

Does the data support the research question or problem statement?

General

Was it an appropriate M/T/IS? Was it explained as such given strengths and weaknesses of the study?

General

Can the results be measured directly? If not, is there a known relationship between measurable variable and the required data? If there is, is it a linear or non-linear relationship?

Variable relationships

What is the error component of the measurement process?

Error

How do you represent non-numerical/qualitative data? How can you show trends in such data etc.? How could it be coded and compared?

Qualitative data

Can a new diagramming technique be developed

New techniques of representing qualitative data

Is bias taken into account? (Hawthorn effect)

Bias

Are techniques other than statistical ones being employed?

Alternative techniques

Will the data analysis yield results that are comparable to other people’s work?

 

If there is triangulation built into the design, if so are the alternative measures compared? Do they support one and another?

Multiple data collection tools

If it is a comparison study, is it a paired or unpaired comparison?

Comparison

If your procedure involves sampling, how was the sample size determined?

Sampling

How has missed data been handled?

Missing data

How are outliers treated? They must be reported, but are they incorporated into the analysis?

Outlying data

Are the results represented in a way that is readily digestible by the readers? (graphically, bar charts, in tables, etc.)

Visual Data Representation – here or in findings or both

If the data is represented graphically, is the correct graphical tool being used?

Visual Data Representation – here or in findings or both

Are the results presented visually in such a way that they can be compared to other researcher’s results?

Visual Data Representation – here or in findings or both

Do the findings extend or delineate the field?

Summary and discussion

Is your technique something that extends the field and can be built upon?

Summary and discussion


Create & Promote New Information

Conduct a mini- data collection /pilot project.

Will your design yield data?
Will the data answer your research question or add to the field?

This page is designed to provide a learning environment for students and lectures of the M.Sc. IT in Education at Trinity College, Dublin. Otherwise, all rights reserved. No part of this work may be reproduced or transmitted in any form or by any means without the written permission of the copyright holder. Such written permission must also be obtained before any part of this work is stored in a retrieval system of any nature.

Contributors of the content of this page include: Bryn Holmes & Damian Gordon so far.

Last updated 31st October, 2000 Comments or corrections should be addressed to:  Bryn.Holmes@tcd.ie