Bert Le Bruyn

Online Teaching Intervention Study @ UU

This blog collects all the updates about OTIS (Online Intervention Study @ UU).

Contacting OTIS

You can reach OTIS by contacting Bert Le Bruyn (b.s.w.lebruyn@uu.nl).

Project aims

Moving from offline to online teaching is a major challenge both for students and teachers. With OTIS, we aim to

• create awareness for (some of) the challenges we’re facing;
• contribute to establishing a dialogue about these challenges;
• document how we’re doing and how we may improve.

You can find a full project overview below but we in particular want to draw attention to the second questionnaire that we have developed.

The second questionnaire

The final version of the second questionnaire is available below. Comments on how it differs from the draft version will be added soon.

The original project description was based on three questionnaires. One at the beginning of block 4, one mid-way and one at the end of the block. In the end, we opted for a setup with two questionnaires, one at the beginning and one at the end. We briefly explain why we made this choice and then present the draft version of the final questionnaire for students. We also comment on the final questionnaire for teachers and the analysis of the questionnaires.

The cancellation of the mid-way questionnaire

The mid-way questionnaire was intended for teachers to see how they could adapt certain choices they had made in the first half of the course. This questionnaire would have been course-specific and would have required intense contact between OTIS researchers and the different teachers.

We decided to cancel the mid-way questionnaire in order not to put too much burden on teachers and students. We further assumed that teachers would make smaller and bigger changes throughout the block. The added value for the course would consequently have been minor.

Draft version of the final questionnaire for students

Below, you can download the draft version of the final questionnaire for students. It is made as parallel as possible to the first questionnaire. The most important changes are the following:

• We limited the introduction.
• We removed general sections.
• We kept the core sections (sense of community, experience with group work, self regulation and motivation) but instead of making them about the period before the Corona crisis set in, we made them about Block 4.
• The sections about sense of community, experience with group work and self regulation were made course specific. This proved hard for the section on motivation because we operationalized it as academic engagement.
• We kept the section on changes in students’ personal situation but made sure that it would be about changes in the course of block 4 that were not mentioned in the previous questionnaire.
• We added a section with open questions about (a) preferred work form, (b) dispreferred work form and (c) general tips and tricks for teachers and students.

The draft version is currently being checked for lay-out and language issues. Suggestions for modification are welcome until Monday June 1st. You can of course also request course-specific changes after that date (in particular if you want to add questions).

Final questionnaire for teachers

There will be a final questionnaire for teachers. We are currently looking into whether this should be in the form of a questionnaire or in the form of a (semi-structured) interview.

Analysis of the questionnaires

Two ICC MA students are working on the analysis of the questionnaires. The last few weeks they have been working hard on their literature reviews and pooling together the data (we received over a hundred questionnaires). Based on the outcomes of the second questionnaire and their literature reviews they will decide which lines of research are the most promising ones and potentially also run follow-up interviews. Their internship papers will present the first results of OTIS. Over the summer we will work on a broader presentation of the data from a more exploratory perspective. Next to the questionnaire data, we will also look at possible correlations between the questionnaire data and the ways the different courses were set up.

The first questionnaire

The questionnaire

The first questionnaire has two versions, one for students and one for teachers. We hope our colleagues and their students will fill it in during or before the first (online) class in Block 4 and use it to initiate a group discussion.

Final versions of the questionnaires are available below (teacher questionnaire will be added on April 19). We also developed a short powerpoint presentation teachers can use to introduce the questionnaire and/or initiate a group discussion:

The different sections of the questionnaires are based on (validated) questionnaires from the literature and on questionnaires that we can compare the outcomes to (in particular the standard course evaluation questionnaire and the questionnaire used for the UU Medewerkersmonitor 2017). Full bibliographical details for the questionnaires from the literature are available below.

Expected time investment for students/teachers

The questionnaire takes about 15 minutes to fill in. We are confident that the time invested is well invested, especially when followed by a group discussion. To accommodate this discussion, the questionnaire comes with an extra tab with ‘group discussion input’ in which the data from the questionnaire have been summarized in an easy-to-use format.

Other than creating an assignment in Blackboard in which the questionnaire can be submitted (teacher) and submitting the questionnaire (in Blackboard for students, to OTIS for teachers), no further action is required from students/teachers until the end of the course if they want to participate in OTIS in a basic way.

Lecture capture

If you’re interested in recording your lectures – for educational purposes or to collect course materials for OTIS – here’s a standard consent form you can use. You can have it submitted through Blackboard.

Overview of OTIS as a research project

The overview below presents the research setup of OTIS and zooms in on five issues: (1) Research questions, (2) Data collection, (3) Expected time investment of teachers/students, (4) Privacy, (5) Output.

1. Research goals

The research goal of this project is to document what is going on in our courses in Block 4. The overarching research question is: how do we deal with a massive move from offline to online teaching?

Three sub-questions will be addressed:

(i) How do we experience online teaching?

(ii) How do we implement our online teaching?

(iii) Do we meet our course goals?

2. Data collection

We collect questionnaire data, course materials and interview data.

Teachers who want to participate with their students can do so by filling in the questionnaire at the beginning and at the end of the block. Extended participation includes (in order of time investment):

• granting us access to course materials (by extension with lecture capture);
• participating in short interviews (mainly end of the block);
• participating in the mid-way questionnaire.

In what follows, we briefly comment on how the sub-questions can be answered with the data we’re collecting. Interactions between the answers to sub-questions (i)-(iii) can be analyzed for courses that go for an extended participation.

2.1. Questionnaires

Questionnaires are developed for the beginning of the block, mid-way and at the end of the block. They allow us to answer sub-question (i).

Both for students and teachers, we’re interested in how they experience online teaching. We measure this experience along a number of dimensions. For students, we look at sense of community, motivation and self-regulation. For teachers, we look at sense of community, work pressure and job satisfaction. We further check a few factors that are likely to be of influence (e.g. previous experience with online teaching).

Questionnaire items are taken from:

Student Questionnaire:

• Fryer, L. K., & Bovee, H. N. (2016). Supporting students’ motivation for e-learning: Teachers matter on and offline. The Internet and Higher Education, 30, 21-29. [constructs computer use and smartphone use]
• Rovai, A. P. (2002). Development of an instrument to measure classroom community. The Internet and higher education, 5(3), 197-211. [construct connectedness (positive wording)]
• Barnard, L., Lan, W. Y., To, Y. M., Paton, V. O., & Lai, S. L. (2009). Measuring self-regulation in online and blended learning environments. The internet and higher education, 12(1), 1-6. [all constructs]
• Brault-Labbé, A., & Dubé, L. (2008). Engagement, surengagement et sous-engagement académiques au collégial: pour mieux comprendre le bien-être des étudiants. Revue des sciences de l’éducation, 34(3), 729-751. [construct academic commitment]

Teacher Questionnaire:

• Fryer, L. K., & Bovee, H. N. (2016). Supporting students’ motivation for e-learning: Teachers matter on and offline. The Internet and Higher Education, 30, 21-29. [constructs computer use and smartphone use]
• Macan, T. H. (1994). Time management: Test of a process model. Journal of applied psychology, 79(3), 381. [construct perceived control of time with provisos from Claessens, B. J. (2004). Perceived control of time: Time management and personal effectiveness at work. Eindhoven: Technische Universiteit Eindhoven.]

We provide stand-alone copies so it’s up to every individual to decide whether they submit it or not. Everyone will further have the option to indicate they agree with their data being used (anonymously) for research purposes or for course purposes alone.

The questionnaire mid-way is optional and can be made part of the course as well and provides relevant information about how everyone’s coping and about the success of teacher/student strategies. To increase the response rate of the questionnaire at the end of the course, we hope teachers will send a personal message to the students to encourage participation.

2.2. Course materials (optional)

We hope teachers will give us access to their course materials. It suffices to grant us access to the online course environments (Blackboard/Teams/…). The materials can then be analyzed by the interns.

The analysis of the materials allows us to answer sub-question (ii). In combination with the questionnaire data, these materials allow us to come to good practices for enhancing sense of community, motivation, etc.

2.3. Interviews (optional)

Next to the questionnaires, we’re also developing interview protocols. We’ll keep these interviews short and we’ll make sure to schedule them when it’s convenient for participants. We hope all teachers and a number of students will take part in an interview on testing and that some teachers and students will also participate in interviews that tackle other issues.

The interviews on testing will allow us to answer sub-question (ii) and can be taken along in the formulation of our best practices.

3. Expected time investment for teachers/students

By integrating the questionnaires into the course, we hope the workload for teachers/students will not be negatively affected.

The interviews, data analysis and data management will be done by interns supervised by Bert Le Bruyn.

4. Privacy

Specifically for students, we propose to collect questionnaires in a decentralized fashion, i.e. per course (e.g. via Blackboard) and to make it clear that the choice of submitting the questionnaires is up to them. In the questionnaires, students can further indicate whether they want to be approached for an interview. Permission for use of data can be withdrawn at any moment by contacting the intern who is responsible for the course.

5. Output

Next to preliminary analyses that can be used by the teachers, the project will lead to a number of internship reports, a final report for university-internal purposes and – potentially – one or more publications.

Bert Le Bruyn