Guarding Against Bias When Training Language Models
MAKING LANGUAGE MODELS DOMAIN-SPECIFIC
The application for which organizations rely on language models is deterministic for negating bias while making language models domain-specific. Here are some best practices for doing so:
Balanced training data: Organizations should represent all the dimensions of a business problem in the training data used to fine-tune models. Take a pharmaceutical company that develops a digital agent, powered by a language model, to answer questions for research. “If the data is not nicely balanced, then you may start seeing responses that you may not realize are what you’re not anticipating as a user,” Osborne remarked. “But, if you look at an evaluation of how your users are interacting with it and the type of responses they’re getting, you may find that they’re very much skewed toward one thing versus another.”
Desired outcomes and points of bias: Organizations should outline desired outcomes for their language model use cases. Then, they should pair this information with ways in which bias could compromise those objectives. According to Allen, “The nature of the bias, I think, is something that is very domain-specific and needs to be understood by the organization adopting the LLM or AI technology. The first thing is to understand your risk points, what you think would be, potentially, ways a model may express bias.”
Data quality: Subsequently, organizations should account for manifestations of bias in the datasets they use to fine-tune models. Initially, they should undertake what Osborne termed “basic data quality measures.” Such measures include ensuring data adheres to proper schema conventions and is complete, current, and accurate.
Testing: It’s also advantageous to find ways to test models for bias when attempting to make them specific for certain domains or business problems. “If it’s consumer-facing, you’d need to test your service in front of a focus group or set of consumers,” Allen recommended. “If it’s internal, you need to test it with a group of internal users.”
TRAINING DATA BIAS
When attempting to reduce the incidence and severity of the bias characterizing language models, organizations should implement a layered approach. The first layer entails diminishing bias prior “to it hitting the model,” Osborne said. This endeavor includes scrutinizing training data for instances of bias, underrepresented facets of a business problem, and overall data quality.
Interestingly, this approach is germane for retrieval-augmented generation (RAG)-based applications involving LLMs too. “We can examine a corpus of documents if you’re setting up a RAG system, for example,” Osborne specified. “Organizations would have control over the data that they’re using as that baseline the model is going to work against. With that in mind, we can explore that data, understand what biases exist in it, and mitigate before we introduce that as another facet to the model.”
If there’s a scarcity of data for certain aspects of a business problem, such as examples of insufficient capital or credit for a digital agent used to influence mortgage decision making, organizations can employ techniques to generate that data.
There is an array of synthetic data approaches that can generate training data so that it’s more balanced. Organizations can also employ language models for such generative purposes. “With unstructured data, it becomes really easy to feel like you can’t change it, you can’t influence it, because it’s so nebulous,” Osborne indicated. “But we do have an opportunity, even using LLMs, to generate information.”