A new research framework called U-SplitDoRA demonstrates how organizations can collaboratively train and adapt large language models without ever exposing sensitive data to a central server or to each other. The approach combines federated learning, where multiple parties train models together, with split learning, which divides computational work between client devices and servers, creating a privacy-preserving system for fine-tuning large language models.

Why Can't Organizations Simply Share Data to Train Better AI Models?

Large language models (LLMs), which are AI systems trained on vast amounts of text to understand and generate human language, require enormous amounts of diverse data to work effectively. However, most organizations cannot share their data due to privacy regulations, competitive concerns, and security risks. Healthcare systems cannot share patient records. Financial institutions cannot expose transaction data. Government agencies cannot reveal citizen information. Yet each of these sectors could benefit enormously from AI models trained on their specific data.

Federated learning solves this paradox by letting multiple data owners train a shared model without ever transferring raw information. Each organization trains the model locally on its own data, then sends only the model updates to a central server for aggregation. However, large language models contain billions of parameters, making them too resource-intensive to train on individual devices, which creates a new bottleneck.

How Does U-SplitDoRA Enable Privacy-Preserving Collaborative Training?

• Three-Part Model Architecture: The framework splits the model into three sections: a head and tail that remain on the client side, and a body that runs on the server, ensuring raw data and labels never leave the organization.
• Weight Decomposition Technique: Instead of using traditional low-rank adaptation (LoRA), a method that reduces the number of parameters needing adjustment, U-SplitDoRA uses DoRA, which updates both the magnitude and direction of model weights, achieving results much closer to full model training while using far fewer trainable parameters.
• Distributed Collaboration: Multiple organizations can participate simultaneously, each training on private data while contributing to a shared global model that benefits from diverse information sources without exposing individual datasets.

Researchers tested U-SplitDoRA using GPT-2-S and GPT-2-M models on the E2E benchmark dataset and found that the framework achieved better accuracy scores and faster convergence speed compared to other state-of-the-art federated learning approaches. This matters because it means organizations can now fine-tune powerful language models for specialized tasks, such as medical diagnosis or legal document analysis, without compromising privacy or exposing confidential information.

What Makes This Framework Different From Previous Approaches?

Earlier split learning frameworks for federated language model fine-tuning, such as SplitLoRA and HSpliLoRA, established the foundation for this research but left critical gaps in privacy preservation and adaptation quality. U-SplitDoRA addresses these gaps by adopting a U-shaped model partitioning strategy that keeps sensitive data on the client side while using DoRA instead of LoRA to improve how well the adapted model performs.

The key innovation is that DoRA updates both the magnitude and direction of weights, whereas traditional LoRA only adjusts direction. This dual approach reduces the performance gap between parameter-efficient fine-tuning and full parameter fine-tuning to a minimal margin, meaning organizations no longer have to choose between privacy and model quality.

The framework is designed for large language models generally, addressing a critical challenge in AI development. As organizations increasingly face pressure to comply with data privacy regulations like GDPR and HIPAA, privacy-preserving fine-tuning methods offer a practical path forward for collaborative AI development without compromising sensitive information.

The research demonstrates that strong privacy protections and effective model performance are not mutually exclusive. By enabling organizations to train powerful AI systems collaboratively while keeping data completely private, U-SplitDoRA opens new possibilities for sectors that have previously been locked out of advanced AI capabilities due to data sensitivity concerns.