Apple's New Foundation Language Models (AFMs) 1. Two Main Models: - AFM-on-device: ~3 billion parameters, for efficient on-device use - AFM-server: Large

Apple's New Foundation Language Models (AFMs) 1. Two Main Models: - AFM-on-device: ~3 billion parameters, for efficient on-device use - AFM-server: Larger model for Private Cloud Compute 2. Architecture and Training: - Based on Transformer with optimizations - Three-stage training: core, continued, and context-lengthening - LoRA adapters for task-specific fine-tuning - Innovative quantization: 3.5-3.7 bits per weight 3. Performance and Benchmarks: - AFM-on-device outperforms larger models (e.g., Gemma-7B, Mistral-7B) - AFM-server competitive with GPT-3.5 - HELM MMLU (5-shot): AFM-on-device 61.4%, AFM-server 75.4% - GSM8K (8-shot CoT): AFM-server 83.3% - Strong in instruction-following (IFEval) - Best overall on Berkeley Function Calling Leaderboard 4. Capabilities: - Excels in instruction following, tool use, writing, math - Long context support up to 32k tokens - Specialized for tasks like summarization 5. Responsible AI: - Focus on user privacy and responsible AI principles - Extensive safety measures (red teaming, human evaluations) - Lower violation rates on safety prompts vs. other models 6. Unique Aspects: - "Accuracy-recovery adapters" post-quantization - Novel RLHF framework: "Iterative Teaching Committee" (iTeC) - New RL algorithm: MDLOO