Corporate Alignment and Safety Filters in Open-Weight AI Models AI-pedia

Corporate alignment and safety filters in open-weight AI models are the behaviours, restrictions and stylistic habits that AI developers train into models whose parameters they release to the public. They include refusing certain requests, adding warnings and disclaimers, avoiding strong positions on contested subjects, and following a company's content policy. In an open-weight model, the learned numerical parameters (the "weights") can be downloaded and run on a user's own hardware. Much of the safety behaviour is stored inside those weights. For this reason it stays with the model even when the model runs offline and no company server is involved.

This produces a situation many users notice and some find frustrating. Running a model locally removes the company's servers, logging and separate moderation systems. The cautious, hedging conversational style learned during training remains. Commentators have described this as a paradox: the computing is decentralised, but the "cognitive priors" are still shaped by a small number of large organisations. Cognitive priors here means the default tendencies and assumptions the model brings to every answer.

The subject sits where several fields meet. These include machine learning research on how alignment works and how easily it can be undone, commercial concerns about legal liability and reputation, national regulation of AI content, and wider debates about whose values an AI system should reflect. It is also tied to the geopolitics of open-weight AI, especially competition between the United States and China.

Terminology

Open weights and open source

According to Wikipedia, open weights are "the publicly released learned parameters of a trained artificial intelligence model, principally its weights and biases". In an artificial neural network, weights are numerical values that determine how strongly inputs contribute to the network's calculations. Publishing them lets others download and use the model. Whether people may modify, fine-tune or redistribute it depends on its licence.

Open weights are distinguished from open-source AI. Under stricter definitions, open-source AI requires public release of more than the parameters. It also covers the source code, training data, evaluation results, intermediate checkpoints and technical documentation. Most widely used "open" language models are open-weight only. Their training data and their alignment datasets in particular are usually not published. As a result, outsiders often cannot see exactly which examples or rules produced a given safety behaviour.

Alignment

In this context, alignment means training a model so that its behaviour matches what its developers intend. That usually means being helpful, being honest and avoiding harm. In practice, "harm" is defined by each developer's policies. The word is also used more broadly in AI safety research for the long-term problem of making advanced AI systems act in line with human values. This article is mainly about the practical, commercial sense.

Safety filters

A safety filter is any mechanism that blocks or changes outputs judged unacceptable. Filters take two broad forms:

• Internal (trained-in) filtering: The model itself has learned to refuse or soften certain answers. This behaviour is part of the weights.
• External filtering: A separate system checks inputs or outputs. Examples are a keyword list, a classifier model or a moderation service. Hosted chatbots commonly use external filters. For open-weight models these are optional, and the person running the model decides whether to use them.

Base, instruct and chat models

A base model has been trained only to predict the next piece of text on a large corpus. It has no particular wish to be helpful and no trained-in refusals. An instruct or chat model is a base model with further training to follow instructions and hold conversations. Most of the alignment behaviour discussed here is added at this stage. Many developers release both versions. Others release only the chat version.

Refusal and over-refusal

A refusal is a response in which the model declines a request, in full or in part. Over-refusal, also called exaggerated safety or false refusal, is a refusal of a request that is actually harmless. A standard example is a model refusing to explain "how to kill a Python process" because of the word "kill". Related behaviours include unrequested moral commentary, repeated disclaimers, and presenting very unequal positions as equally supported, which critics call "false balance".

Background and history

From base models to aligned assistants

Researchers developed reinforcement learning from human feedback (RLHF) in the late 2010s. A widely cited early paper was published in 2017 by Paul Christiano and colleagues at OpenAI and DeepMind. In 2022, OpenAI described the InstructGPT models. These applied RLHF to make GPT-3 follow instructions and reduce harmful outputs. ChatGPT, released in November 2022, used a similar approach. It established the now-familiar style of a polite assistant that declines some requests and often adds caveats.

The open-weight wave of 2023

In February 2023, Meta released the first LLaMA models to researchers on request. The weights were soon shared widely online. Over the next months, academic and hobbyist groups produced instruction-tuned versions cheaply. Stanford's Alpaca was trained on instruction examples generated by an OpenAI model. Vicuna was trained on user-shared ChatGPT conversations. These projects showed a pattern that became common: open models learned conversational behaviour, including hedging and refusals, by imitating outputs from proprietary systems.

Some users objected to the inherited refusals. "Uncensored" fine-tunes appeared, made by removing refusal and moralising examples from the training data before fine-tuning. A widely discussed early example was a series of "uncensored" WizardLM variants released by the developer Eric Hartford, who also argued publicly for the approach.

In July 2023, Meta released Llama 2 with an official chat version, Llama 2-Chat, aligned with RLHF. Many users and researchers found it very prone to over-refusal. It was often cited as an example of safety tuning generalising too broadly. Meta's reference materials at first suggested a default system prompt focused on safety. This made the over-refusal worse, and Meta later stopped recommending it.

In September 2023, the French company Mistral AI released Mistral 7B under the Apache 2.0 licence. It had little trained-in moderation. This drew both praise and criticism. Mistral later documented an optional system prompt for enforcing guardrails, which put the choice in the deployer's hands.

Separate guard models

In December 2023, Meta announced Purple Llama, a set of open safety tools. It included Llama Guard, a language model fine-tuned to classify prompts and responses against a list of harm categories. Google later released ShieldGemma for its Gemma family. These guard models reflect a different idea of safety for open weights: the main model can be made less restrictive, and policy enforcement can be a separate, replaceable component.

Chinese open-weight models

From 2023 onward, Chinese developers released a growing number of capable open-weight models. These included Alibaba Cloud's Qwen series, DeepSeek's models, Moonshot AI's Kimi models and Z.ai's GLM models. They added a second, different set of trained-in restrictions. Chinese regulations require generative AI services to follow state content rules. Users found that some of these models refused or deflected questions on subjects that are politically sensitive in China. DeepSeek's R1 reasoning model, released in January 2025, drew wide attention for this behaviour as well as for its capabilities.

Current landscape

Wikipedia describes the open-weight field as being shaped by US–China competition, sometimes called an "AI arms race" or "AI Cold War". It states that Chinese companies such as DeepSeek, Alibaba Cloud, Moonshot AI and Z.ai mostly use an open-weight approach under permissive licences like Apache or MIT. US companies, which it lists as OpenAI, Anthropic, Google DeepMind, SpaceXAI and Meta Superintelligence Labs, favour proprietary approaches, especially for their largest models. Wikipedia notes that these are partly national policy matters and that some US politicians have called for restricting US public access to Chinese AI tools.

There are exceptions. In August 2025, OpenAI released two open-weight models, gpt-oss-120b and gpt-oss-20b, under the Apache 2.0 licence. It published safety testing that included deliberately trying to fine-tune the models toward harmful capabilities before release. According to Wikipedia, as of July 2026 the largest open-weight frontier model is Moonshot AI's Kimi K3 at 2.8 trillion parameters, followed by Alibaba Cloud's Qwen3.8 at 2.4 trillion. Open-weight releases outside China are led by the US lab Thinking Machines Lab, Nvidia's Nemotron family and France's Mistral AI. These figures come from the source article and are reported here as stated there.

How alignment is built into the weights

Developers use several training stages. Each can affect safety behaviour.

Pretraining data filtering

Even before alignment, developers usually filter pretraining data. They remove material such as spam, duplicated text, personal data and certain categories of harmful content. This shapes what the base model knows. It can make some outputs less likely or impossible. It does not usually produce the conversational refusals that users notice.

Supervised fine-tuning

In supervised fine-tuning (SFT), the model is trained on example conversations that show the desired behaviour. If those examples include refusals, warnings or particular phrasings ("It's important to note that..."), the model learns to produce them. SFT data may be written by people, generated by other models, or both.

Reinforcement learning from human feedback

In RLHF, human annotators compare pairs of model responses and pick the better one according to a written guideline, often called a rubric. A separate "reward model" is trained to predict these preferences. The main model is then optimised to produce responses the reward model scores highly. Anything the annotators consistently prefer or reject becomes a strong tendency. This includes caution on controversial topics.

Direct Preference Optimization

Direct Preference Optimization (DPO) was introduced by Rafael Rafailov and colleagues in 2023. It uses the same kind of preference comparisons but skips the separate reward model. The model is updated directly to favour preferred responses over rejected ones. DPO and related methods became popular for open-weight models because they are simpler and cheaper than full RLHF.

AI feedback and written principles

Some methods replace part of the human labelling with judgements from AI models. In 2022, Anthropic described "Constitutional AI". In this method a model critiques and revises its own outputs against a written list of principles, and AI-generated preferences are then used for training. The general approach is often called reinforcement learning from AI feedback (RLAIF). It makes the developer's chosen principles more explicit. It also means one company's model, and its habits, can strongly shape another model's training.

Synthetic data from proprietary models

Many open-weight models have been fine-tuned on synthetic data, meaning text generated by other AI models. The best-known proprietary assistants, such as OpenAI's GPT-4 family and Anthropic's Claude, were common sources, especially in 2023 and 2024. Students imitate teachers, so open models trained this way can pick up the teacher's style. This can include its disclaimers, structured hedging and tendency to give "both sides" answers. Commentators sometimes call this a synthetic-data "echo chamber". The extent varies by model and is hard to measure without access to the training data. Many proprietary providers' terms of service prohibit using their outputs to build competing models, but enforcement has been limited.

Chat templates and default system prompts

Chat models are trained with a fixed format, called a chat template, that marks system instructions, user messages and assistant replies. Some models come with a recommended default system prompt. A system prompt is a hidden instruction placed before the conversation. These defaults can add caution on top of what the weights already contain. The user can change or remove them when running a model locally.

Separate safety filters and guard models

Hosted AI services usually combine an aligned model with external moderation. A provider may check user input before it reaches the model and check the model's output before showing it. When the weights are released, none of these external layers come with them unless the deployer chooses to add them.

To fill this gap, several developers have released dedicated classifier models:

• Llama Guard (Meta): a fine-tuned Llama model that labels a prompt or response as safe or unsafe and names the violated category. Several versions have been released, including ones that handle images.
• Prompt Guard (Meta): a small classifier aimed at detecting prompt injection and "jailbreak" attempts. A jailbreak is a prompt designed to get around a model's restrictions.
• ShieldGemma (Google): classifiers built on Gemma for categories such as harassment and dangerous content.

Supporters argue that separating the filter from the model has two benefits. Businesses can apply strict policies where needed. Researchers and individual users can run the base or chat model without them. Critics note that guard models make their own classification errors, including false positives on harmless text.

Why local models still behave cautiously

The runtime is neutral; the weights are not

Local inference tools such as llama.cpp and Ollama only perform the calculations needed to run a model. Ollama was originally built as a user-friendly layer on top of llama.cpp. These programs do not add refusals or opinions. When a locally run model hedges or refuses, the behaviour comes from its weights, its chat template or its system prompt. It does not come from the software running it.

Liability and reputation

Companies that release models stay publicly associated with them. A model that produces offensive, dangerous or defamatory content can cause negative press, legal risk or regulatory attention, even when someone else runs it. This gives developers a reason to train conservative behaviour into the weights before release. They cannot rely on the downstream user to apply filters.

Licences and acceptable use policies

Many open-weight licences include acceptable use policies that prohibit certain applications. The Llama community licence and Google's Gemma terms are examples. Trained-in caution is partly a technical expression of these policies. Some other models are released under permissive licences such as Apache 2.0 or MIT, which do not have such policies.

Who designs the preferences

The preference rubrics that guide RLHF and DPO are written by the developing company. Critics argue that these rubrics reflect the cultural and political assumptions of a narrow group, often described as a Silicon Valley corporate culture. They say this leads to answers that touch sensitive cultural subjects being marked down. The picture is mixed. The people who actually do the labelling are often contractors hired through outsourcing firms in many countries. OpenAI's InstructGPT paper, for example, reported labellers of several nationalities. Even so, the guidelines that labellers follow are set centrally by the developer. That is the main reason critics describe the resulting values as corporate rather than broadly representative.

Overgeneralisation of safety training

Safety training gives a model examples of things to refuse. The model does not necessarily learn the reasons behind those examples. Instead it may learn surface features, such as particular words, topics or tones that predict a refusal. As a result, prompts that mention politics, religion, history, crime, health or social issues can trigger lecturing, false balance or outright refusal, even when the request is harmless. Researchers describe this as a trade-off between harmlessness and helpfulness.

National regulation

Some restrictions reflect legal requirements in the developer's home country, not corporate preference. The clearest example is the political content restrictions in Chinese models, described below.

Over-refusal and its measurement

Researchers have built benchmarks to measure how often models refuse harmless requests:

• XSTest (Paul Röttger and colleagues, 2023): a set of safe prompts that superficially resemble unsafe ones, paired with truly unsafe contrast prompts. It tests whether a model can tell them apart. Llama 2-Chat showed high rates of exaggerated refusal in the original study.
• OR-Bench (2024): a larger benchmark of "seemingly toxic" prompts, made to measure over-refusal across many models. Its authors reported a strong correlation between how safe models were and how often they over-refused. This suggests the trade-off is hard to avoid with current methods.

Over-refusal is not only a matter of complete refusals. Partial refusals, warnings attached to a correct answer, and unrequested ethical commentary are harder to count. Benchmarks measure them less consistently. For users who prefer direct and literal communication, such additions can make answers longer and harder to use, even when the core content is correct. This includes many autistic users.

In general, newer model generations from major developers have been tuned to refuse less than early 2023-era chat models. Developers have publicly cited over-refusal as a problem they wanted to reduce. How far this has worked is still debated and differs from model to model.

Regional and political dimensions

Content requirements in China

China's Interim Measures for the Management of Generative Artificial Intelligence Services took effect in August 2023. They require public-facing generative AI services to follow content rules, including adherence to "core socialist values", and to undergo security assessments. Models from Chinese developers often refuse or redirect questions on politically sensitive subjects. Examples include the 1989 Tiananmen Square protests and the status of Taiwan. Analysts have found that some of this behaviour lives in the weights and persists when the model runs locally. Other parts of it are applied only by the developers' hosted services.

In February 2025, the US company Perplexity released "R1 1776", a version of DeepSeek R1 post-trained to answer questions it judged to have been censored. This shows that restrictions in open weights can be modified by third parties. It also shows that "removing censorship" involves the modifier's own judgements about what counts as accurate.

Political lean and value disagreement

Several studies have tested language models with political orientation questionnaires. Some, including work by the researcher David Rozado, reported that many chat models give answers that score left of centre on such tests. Critics question these methods. They argue that multiple-choice questionnaires do not reflect how models behave in open-ended use, and that results change a great deal with small changes to the prompt. Researchers also disagree about how much of any measured lean comes from pretraining data and how much from alignment tuning.

More generally, any choice about which topics are sensitive, which views get a disclaimer and which requests are refused is a value judgement. Different countries, communities and individuals make these judgements differently. This is a central reason why some people see alignment as a form of corporate or state governance of speech, not a neutral technical step.

Removing or modifying alignment

Open weights can be changed, so users and researchers have developed several ways to reduce or remove trained-in restrictions.

System prompts

The simplest method is a system prompt that asks for a different style. An example is an instruction to answer plainly and directly, without unrequested moral disclaimers or caveats. This often reduces hedging a great deal. It changes nothing in the weights, and strongly trained refusals may remain.

Fine-tuning

Further fine-tuning can change a model's behaviour substantially. Studies in 2023 showed that safety training in open models can be largely undone cheaply:

• Work by Xiangyu Qi and colleagues found that fine-tuning aligned models on even a small number of harmful examples could break their safety behaviour. Fine-tuning on harmless data could also weaken it unintentionally.
• Other research projects, including "BadLlama" and a study using LoRA on Llama 2-Chat, showed that refusals could be removed with modest computing budgets. LoRA (low-rank adaptation) is a cheap fine-tuning technique that trains a small set of extra parameters.

Community "uncensored" models usually take a gentler approach. They fine-tune on instruction datasets from which refusals and moralising have been removed. They do not train on harmful content.

Abliteration

In 2024, Andy Arditi and colleagues published "Refusal in Language Models Is Mediated by a Single Direction". They found that in many chat models, refusal behaviour corresponds largely to one direction in the model's internal activation space. Specifically, this is in the residual stream, the running internal representation that passes between the model's layers. Adding this direction makes the model refuse harmless requests. Removing it largely stops the model from refusing harmful ones.

Hobbyists turned this into a technique called abliteration, a blend of "ablation" and "obliteration". The refusal direction is estimated by comparing activations on harmful and harmless prompts. It is then removed from the weights by orthogonalisation, a linear-algebra operation that makes the weights unable to write along that direction. No retraining is needed. Abliteration is a form of representation engineering, meaning methods that read or edit a model's internal representations directly. Many "abliterated" versions of popular models have been shared on Hugging Face, often in the GGUF file format used by llama.cpp and Ollama.

Limits and side effects

• Abliteration and uncensored fine-tuning can slightly lower performance on other tasks. Users often report this, and some evaluations confirm it, though the size of the effect varies.
• Removing refusals does not necessarily remove a preachy style. Hedging phrases and both-sides framing may be stored in different ways from outright refusal.
• Removing refusals also removes protection against truly harmful requests. This is the central concern of critics of the practice.
• Modified models carry their modifiers' choices and may introduce new biases.

Robustness and tamper-resistance research

Because trained-in safeguards are easy to remove, researchers have looked for methods that survive fine-tuning. The approaches include:

• Tamper-resistant training: methods such as TAR (Tamper-Resistant Safeguards), described in a 2024 paper, train a model so that later fine-tuning struggles to restore specific harmful capabilities. Independent evaluations have questioned how robust such methods are against determined attackers.
• Pretraining data filtering: removing certain hazardous technical knowledge before pretraining, so the model never learns it. A 2025 study by EleutherAI and the UK AI Security Institute reported that this kind of filtering resisted later fine-tuning better than post-training safeguards.
• Pre-release adversarial testing: deliberately fine-tuning a model toward misuse before release, to estimate the worst case. OpenAI reported this approach for gpt-oss.

These methods mostly target catastrophic risks, such as help with biological or chemical weapons. They do not target the everyday caution and hedging that most users complain about. Many researchers distinguish between these two kinds of safety behaviour.

Policy and regulation

• United States: In July 2024, the National Telecommunications and Information Administration (NTIA) published a report on dual-use foundation models with widely available weights. It recommended against restricting open weights at that time and called for monitoring. California's SB 1047, which would have imposed safety obligations on developers of very large models, was vetoed in September 2024. Some politicians have proposed limiting access to Chinese AI models.
• European Union: The AI Act gives partial exemptions to general-purpose models released under free and open-source licences. The exemptions do not apply to models classed as posing "systemic risk", a category linked mainly to very large training computation.
• China: Generative AI services must comply with the 2023 interim measures and related content rules, as described above.

The policy debate often treats alignment in open weights as a national-security question. It focuses on whether models could help with weapons or cyberattacks. The cultural and stylistic side of alignment draws more public complaint but less regulatory attention.

Debates and open questions

• Safety versus control of speech: Supporters see trained-in safeguards as responsible practice, especially once weights cannot be recalled after release. Critics see them as private companies or governments deciding what people may ask a tool they run on their own computer.
• Whose values: It is unresolved whether a single set of preferences can suit users across cultures, or whether models should be steerable to many value systems. Proposals include greater "steerability", published behaviour specifications, and community-designed alignment.
• Real risk versus symbolic caution: Many researchers argue that refusing information that is freely available elsewhere adds little safety. Others argue that even marginal reductions in misuse matter.
• Can safeguards survive in open weights? Current evidence suggests post-training safeguards are easy to remove. It is uncertain whether tamper-resistant methods will become reliable.
• Transparency: Most developers do not publish their alignment data or full rubrics. This makes it hard to audit the origin of particular behaviours or to separate corporate policy from legal requirement.
• Synthetic data loops: As more training data is produced by aligned models, some observers worry that particular styles and blind spots could become entrenched across the whole field.
• Measurement: Over-refusal, preachiness and political lean are hard to measure objectively, and results depend heavily on the prompts used.

See also

• Open weights
• Open-source artificial intelligence
• Large language model
• Reinforcement learning from human feedback
• Direct Preference Optimization
• Constitutional AI
• AI alignment
• AI safety
• Prompt engineering and jailbreaking
• Representation engineering
• Llama (language model)
• Qwen
• DeepSeek
• Mistral AI
• llama.cpp and Ollama
• Hugging Face
• Artificial Intelligence Act (European Union)
• Internet censorship in China
• Algorithmic bias