MoE models

2 items

RESEARCHarXiv CS.CL·5d ago

Expert-Aware Refusal Steering

This paper extends refusal steering to Mixture-of-Experts (MoE) Large Language Models, finding that steering performance is not hindered by the MoE architecture. It proposes expert-aware refusal steering methods that leverage expert routing patterns, demonstrating that refusal behavior can be effectively steered based on a single expert's output.

MoE models inference refusal steering AI alignment

RESEARCHarXiv CS.LG·7d ago

BitsMoE: Efficient Spectral Energy-Guided Bit Allocation for MoE LLM Quantization

BitsMoE proposes a spectral-energy-guided bit-allocation framework for quantizing Mixture-of-Experts (MoE) large language models. It addresses memory-intensive deployment by decomposing MoE layers and using expert-specific spectral factors for fine-grained, activation-aware mixed-precision quantization.

MoE models deep learning AI optimization quantization