User Modeling in Educational
Domain

Maroš Unčík
supervisor: prof. Mária Bieliková

Adaptive Systems

The trend of using e-learning systems is progressively growing and opportunities that the Web 2.0 provides are huge. Nowadays, e-learning systems offer more and richer content, enable communication and collaboration among users. The rise in use of these systems causes information overload. The adaptive e-learning systems are trying to address the most crucial issues, which are related to this overflow: (1) adaptive systems present only information, which is appropriate and/or interesting for him/her at the moment, (2) help user to choose the way to proceed when viewing content, (3) prevent user from getting lost in the content and/or avoid to prevent him/her from forgetting the original objectives.

A lot of problems in the domain of user modeling were identified. Combinations of several different inputs entering the user modeling process or the use of information about user beyond the adaptive system to enrich the user model are just two of them. Another challenge, the scrutability, concerning the visibility of the user model to users, is also closely related. In most of the systems, the user cannot directly access the user model and cannot provide explicit feedback about him/her, which could be otherwise taken into account. Much work has been devoted to resolve these problems.

In our work we deal with an user model and its visualization. We consider the visualization as a tool for the user model evaluation. Another aspect is to use the visualization of the user model from user’s point of view, which allows direct and explicit feedback from students to enrich the user model. The visualization can also answer questions about what the system believe to be true, what it believes to be false and to find relationship between these beliefs, if they exists. It brings also another benefits, since many real-world user models are likely to be large, the visualization helps the user:

  1. to get an overview of the whole model,
  2. to get a clearer overview of dependencies in the user model, and
  3. to adjust the sensitivity of the user model.

User Model

The heart of any adaptive system is an user model. For this reason, we design an overlay user model based on light weight representation. In the proposed user model, built above the domain model, we use domain-independent (e.g. age) and domain-dependent characteristics (knowledge and interests). The domain-dependent characteristics characterize a user in relation to concepts of the domain model. The value of the domain-dependent characteristics is represented by a three-dimensional vector [level, confidence, source]. The level of the characteristic is in fact its value. It takes real values from the closed interval <0, 1>, where 1.0 denotes the maximum value. Each characteristic is associated with the confidence which expresses probability that the user really has given value. Each characteristic is determined by a source, which can be either a tool or a method. Characteristics of knowledge and interests are directly changed by the behavior of the user, based on recorded activity in the educational system. To determine these characteristics, we can use several sources. The problem is that every single characteristic has more probability-value pairs from different sources. For practical reasons we need only one value-confidence pair, which characterize the relation between user and concept. Therefore, each pair must be appropriately combined. To this purpose we designed method for combination more inputs, which enter the user model.

Pridaj nejaké tie informácie a obrazok rovnic

Visualization

As a tool for user model evaluation we design a visualization. It allows to interfere between believes modeled by the system and what thinks user. We assume three main features:

  1. to show the student what is the system believes about her,
  2. to provide an insight into a user model and to show its time changes, and
  3. to allow students to give explicit feedback.

We decided to visualize the user's knowledge as a graph. Vertices in the graph represent domain relevant terms and links among them represent relationships represented in the domain model. Each term has connections to other term if there is relationship between them. Such visualization is sufficiently comprehensive, but easy to understand, to show the relationships between concepts and to show a way they interact.

Another important aspect is the color scale we used in the graph. We use levels of green, red and white color. The green color is used to label the terms that user has understood. The brighter color indicates the level of user knowledge about the term. The red color shows the term with some sort of defect in the learning process. For example, if the user has not understood the term, the hue of red color is stronger. If the user has not visited the term yet, we do not have any information about the level of knowledge for this term, we use the white color.

The basic overview shows all relevant domain terms and all relations among them. The user may select a particular terms, which is she interested in. In this case, the selected term is highlighted in yellow and visualization shows dependent and associated term to selected terms. The user can add more details with a decreasing of a slider in the left. The closest visualization shows only prerequisites and related concepts of the first level.

An important part of visualization is an explanation for the user, which is represented by a natural language form of pop-up window, which appears while pointing to the concept. The explanation gives basic information about knowledge achieved by user. If the user wants to know more specific details, she can use the "I want to learn more" option, which allows seeing more detailed time changes. The explanation is generated on the basis of information, we have logged about a user. Thus, we can determine which learning objects the user was working with. We use the weight of the relation concept-to-learning-object from a domain model to show to the user only relevant learning objects related to chosen concept.

Shorterm Experiment with Visualization

We experimented with the visualization. The participants were five university students, who are not studying in field of informatics, but they had some experience with computers. Each student had to explore the same example of the visualization of a user model. The example was taken from domain of English course. We show the example of visualization to students and a scenario of evaluation was following:

  1. We provided to participants the example of visualization and following explanation: Imagine an electronic e-learning system for learning English. In such system there is a component, which shows your overall knowledge. Example of such a component is shown in the following figure below. Imagine that it is your overall knowledge. Please answer the following questions.
  2. We asked Question 1 and Question 2.
  3. After users provided responses to the Question 1 and Question 2, we explained idea of visualization. We explained them the color scale and dependencies among components. After that, we asked the Question 3.

The questions were as follows:

  1. Look around the figure and comment whether you think the components on figure denotes.
  2. What represent the colors in the figure?
  3. What do you think of the idea to show people information about their overall knowledge?

This experiment suggested that the participants considered the visualization of user model interesting. They appreciated the general principle of the visualization, and understood the proposed visualization. In addition, although a Participant 3 made negative comments about aspects of the interface, others appeared to be able to figure out the main purpose of such visualization without explanation and they appeared to be able to use it effectively in real conditions.

User Model Experiemnt

We verify the user model in domain of Functional and Logic Programming in the second semester of the academic year 2011/2012, which takes place in graduate study at FIIT STU. The course consists of two parts. In the first part of the semester, students study functional programming language Lisp, in the second part of the semester students study logic programming language Prolog. We used system ALEF [8] for evaluation. The advantage of using this system for verification was that students were familiar with the system and have access to the same learning materials during the experiment, as in the actual teaching of the subject. It was also beneficial to students during the semester working with the system ALEF, the system captures information about actions performed by users in the system. Based on this information we are then able to generate for each user the current user model, which we used in the experiment.

Component for user model verification was displayed below each learning object. In this component we show participants two different task:

  1. task: We shown participants the concepts that have been related to displayed learning object. The concepts were not highlighted in color and participant of the experiment had to select two concepts they think that they knew best.
  2. task: We shown participants the visualization of their own user model. Participant was asked about a particular concept of visualization. The question was: "Do you think that the information about the concept XXX is correct?" where XXX is a specific concept. Possible answers were yes, no or I do not know.

In the experiment participated 6 students. They responded 143 times to first task and 129 times to second task. Together, they resolved 272 tasks. Besides this, participants could provide their ideas and comments in component for user model evaluation. We gained 15 of these.

Task 1

From the 143 replies we received, we determined the precision of selected concepts. We checked whether the participant actually chose two of the best concepts from concepts offered. Results can be seen in the table 4. At the same time if the user did not choose one of the two best concepts we offered, we identified the two tolerances, when we considered selected concept as correct. The first possibility, we have tolerated was if concept was from first quarter.

Task 2

In the task 2 participants had to decide whether the claims that the user model displayed are correct. The gained data included information on the displayed visualization and participants responses. The answers are presented in Table 5. From the received responses we can see that only in 97 cases out of 129 (80% responses), users think that the arguments which was shown in the visualization was correct. Only 9% of cases, users identify the statements in the visualization are not correct.

Conclusion

The experiments suggested several conclusions. Visualization of the user model in web support for education is interesting for users. Surprising is, that some users perceive the visualization of the early work with educational system as a distraction, because by the beginning of their work it could act as a deterrent. It is therefore appropriate to show such users visualization of model after they have gained certain level of knowledge and have initialized the user model.

In the experiments, we achieved relatively good results in the evaluation of user model. The favorable results could take up the fact that the experiment was performed at the end of the semester when the students' knowledge of the subject is high. Therefore the level of knowledge in user model was high as well.

The main drawback of experiments was that just a few people have been involved in and also the duration of the experiment. To achieve more accurate results, it would be useful if the experiment could attend more parties, who worked with visualization at home for a longer period of time - several weeks, but preferably the entire semester. This would allow us to track users' knowledge in educational system from the beginning. This would result in the elimination of conditions of controlled experiment, and users would have behaved in more natural way. On the other hand in a controlled experiment we have noticed some behavior that the users do not realize when using applications, respectively they make them different. This allowed us to find a contradiction in what users respond to questions and what is the real behavior.

About Author

Maroš Unčík studied at Faculty of informatics and information technologies at Slovak University of Technology in Bratislava. He succesfully finished his studies and gained master degree in Software engeneering in June 2012.











Contact: uncik07[at]fiit.stuba.sk