AI misinterpretations and bias can be roughly divided into:
- those caused by AI (e.g. algorithmic and data bias),
- those caused by user actions or interpretations (e.g. lack of skills or misinterpretation of the information generated by AI), and
- systemic, meaning those caused by the operation of the education system and the operating culture of the organisation (e.g. the structures and actors of the education system may affect the data used and the processing of the data, causing bias).
It should also be noted that the development of AI applications is not isolated from the cultural and historical contexts, values and other underlying factors. At present, AI systems in general and generative AI in particular are largely based on the commercial development efforts of large companies, the data that they use and the cloud computing capacity of the cloud servers that they operate. As a result, AI applications rarely take into account factors such as the special features and cultural context of the Finnish language or the value base of the Finnish national curricula.
Factors that affect bias, misinterpretations, and risks of discrimination in the use of AI include
Errors in algorithms or the design of AI models
Design errors in the development of algorithms or AI models may distort algorithms and the interpretation of data, even if the actual data used is accurate. The errors can include both logical design errors and minor technical errors. The development of algorithms, the training of AI, and the testing and post-processing have a major impact on whether algorithms and AI operate without bias, and whether these may result in negative impacts on equality, such as discrimination.
Using data in the wrong context
The data used to train AI models can include data sets or variables that are not suitable for the application. The data may also not be expressive enough to describe the desired phenomenon, or data may simply be used in the wrong context or be based on poorly chosen variables. For example, learning analytics data that contains log data on sign-ins, clicks and time spent in an online learning environment is poorly suited for creating AI models on learners' motivation or forecasting learning outcomes.
Insufficient data
Not having enough data to train AI models on can prevent the creation of high-quality and accurate AI models and the testing thereof. This can easily lead to AI misinterpretations and bias. The creation of machine learning models often requires very large data sets, known as big data.
Selection or sampling bias
The data used to train an AI model may have been poorly selected, one-sided or poorly suited to the purpose of the application. In this case, the data may not be representative of larger groups and their properties, creating bias in the AI model and thus its outputs.
Errors and missing values in the data
Data sets may often contain invalid or missing values. If the data is erroneous, the AI’s output cannot be valid and reliable. Data cleaning and quality assurance is thus the most important stage in AI development. Different methods can be used to correct missing and abnormal values, the reliability of which must be evaluated separately.
Faulty generalisation
The use of AI and AI models can easily lead to faulty generalisations between populations. Especially in learning analytics, the fact that you cannot draw direct conclusions between populations must be taken into account. For example, it is not possible to draw very strong conclusions about class 5B based on training data from class 5A. On the other hand, the learning outcomes of last year’s class 5B cannot be generalised to this year’s class 5B.
Aggregation
If you use heterogeneous data to calculate averages or create models based on averages, for example, they will no longer describe the target group of the data very well, nor can they be applied to individual learners, for example. Instead of group-based calculations (including the Gaussian curve), using an individual's long-term data, the so-called idiographic approach, often provides more accurate results in AI and learning analytics.
Outdated data
The data used to train the AI model may describe past events, situations and structures and thus be ill-suited to explain the current context. For example, previously collected data may reflect attitudes, disparities, or gender inequality that have existed — or continue to exist — in society, including unequal treatment based on grounds of discrimination. On the other hand, when the educational system is changed and curricula are reformed, it is necessary to carefully assess the suitability of previous data on learners to the current situation. The key question is how well can we predict the future based on history? In addition, it should be noted that the legal obligations to promote equality and non-discrimination require organisers of education and early childhood education to actively eliminate inequality. One way to promote equality and non-discrimination is through positive special treatment.
In addition to AI bias, there is also sometimes talk of AI hallucinations, which means an AI making up things that are not actually real or true. In most cases, hallucinations are not actually caused by bias in the data or algorithms, but rather by the AI generating output based on its language model and data that is correct on the basis of the language model and data, but inconsistent with reality.
In the equality auditing and testing of artificial intelligence models, it is essential to take into account the starting points and obligations set out in national equality and non-discrimination legislation. The methods used internationally may not necessarily align with national equality and non-discrimination regulations.