[{"data":1,"prerenderedAt":392},["ShallowReactive",2],{"content-\u002Fposts\u002Fwhy-llm-seems-worse-after-launch":3,"blog-posts":140},{"id":4,"title":5,"body":6,"categories":125,"date":126,"description":127,"draft":128,"extension":129,"meta":130,"navigation":131,"path":132,"seo":133,"stem":134,"tags":135,"__hash__":139},"posts\u002Fposts\u002Fwhy-llm-seems-worse-after-launch.md","为什么模型上线后，体感像是“降智”了？",{"type":7,"value":8,"toc":117},"minimark",[9,13,18,29,50,59,63,66,74,77,80,88,91,94,97,100,103,114],[10,11,12],"p",{},"如果你觉得某个模型刚上线时表现更好，但过一段时间后，同一个模型名下的回答却像变迟钝了，serving 侧的量化值得排查。量化能降低显存占用和推理成本，也可能改变模型输出。",[14,15,17],"h2",{"id":16},"checkpoint-与-serving-配置","Checkpoint 与 serving 配置",[10,19,20,21,25,26,28],{},"仅知道训练得到的 checkpoint，并不能判断线上服务是否直接按 checkpoint 保存时的精度运行。Serving 配置涉及模型计算所用的 ",[22,23,24],"code",{},"dtype","、权重量化方案和 KV cache 的存储 ",[22,27,24],{}," 等方面；这些配置能否组合使用，以及量化是否同时覆盖权重与激活，取决于量化方案、模型和硬件。",[10,30,31,32,35,36,43,44,49],{},"以 vLLM 为例，在线量化可以在加载 BF16\u002FFP16 模型时把 Linear、MoE 权重转换为 FP8；KV cache 也有独立的 ",[22,33,34],{},"--kv-cache-dtype"," 设置。",[37,38,42],"a",{"href":39,"rel":40},"https:\u002F\u002Fdocs.vllm.ai\u002Fen\u002Flatest\u002Ffeatures\u002Fquantization\u002Fonline\u002F",[41],"nofollow","vLLM 在线量化文档","、",[37,45,48],{"href":46,"rel":47},"https:\u002F\u002Fdocs.vllm.ai\u002Fen\u002Flatest\u002Fcli\u002Fserve\u002F",[41],"serving 参数文档"," 展示了这条技术路径，但没有说明任何特定厂商怎样部署。",[10,51,52,53,58],{},"因此，量化不一定以替换 checkpoint 文件的方式发生；服务可以从上线之初就使用量化配置，也可以之后调整配置。对外模型名本身无法告诉用户后端使用的精确 revision、路由或数值格式；“上线时直接跑训练 checkpoint，后来才换成量化 checkpoint”只是众多可能部署过程中的一种。",[37,54,57],{"href":55,"rel":56},"https:\u002F\u002Fdocs.vllm.ai\u002Fen\u002Flatest\u002Fapi\u002Fvllm\u002Fconfig\u002Fmodel\u002F",[41],"vLLM 模型配置文档"," 也分别配置了对外服务名与模型 revision。",[14,60,62],{"id":61},"权重量化与-kv-cache-量化","权重量化与 KV cache 量化",[10,64,65],{},"权重量化用较少位元表示参数，能压缩权重显存占用，并减少解码时读取权重的内存流量；端到端延迟是否下降，取决于量化内核和硬件能否高效执行。数值近似会改变前向计算得到的 logits。如果最高分 token 与其他候选 token 接近，细微误差就可能改变下一个 token，自回归解码随后会沿着新的文本继续。输出因此可能变化，任务质量却不必然明显下降。",[10,67,68,73],{},[37,69,72],{"href":70,"rel":71},"https:\u002F\u002Farxiv.org\u002Fhtml\u002F2210.17323v2",[41],"GPTQ（GPT 模型后训练量化方法）论文"," 在 OPT（Open Pre-trained Transformer）、BLOOM（BigScience Large Open-science Open-access Multilingual Language Model）等模型及 perplexity、zero-shot 基准上评测了离线权重量化。",[10,75,76],{},"例如，OPT-175B 在 WikiText2 上的 perplexity，FP16 基线为 8.34，4-bit GPTQ 为 8.37。该结果表明这一方案在这个指标上保留了质量，但不能保证其他量化方法、模型或线上对话也有相同表现。论文报告的速度收益依赖定制 GPU kernel 和减少内存搬运的实现。",[10,78,79],{},"权重之外，推理还会保存已处理 token 的 key\u002Fvalue，供后续 token 生成时计算 attention。活动序列越长、并发请求越多，KV cache 通常占用越多显存，因此压缩它能腾出并发空间。KV cache 精度与权重精度是不同的配置维度，调整 KV cache 精度也可能影响回答。",[10,81,82,87],{},[37,83,86],{"href":84,"rel":85},"https:\u002F\u002Farxiv.org\u002Fhtml\u002F2402.02750v2",[41],"KIVI（KV cache 非对称量化方法）论文"," 给出了一个 2-bit KV cache 方案：key 按 channel 量化，value 按 token 量化，并为近期 token 保留全精度残余区。它在 Llama、Mistral 的若干基准上报告较小的质量下降，也评测了 LongBench 长上下文任务；论文同时指出，Falcon 的多查询注意力让原始 KV cache 较小，2-bit 方案在部分任务上可能明显退化。",[10,89,90],{},"论文摘要报告，Llama-2-7B 使用 KIVI 时的峰值总显存约为 FP16 基线的 1\u002F2.6（统计口径包含模型权重）。",[10,92,93],{},"第 4.2.4 节另报告一组吞吐实验：研究者在单张 80GB A100 上使用 ShareGPT 衍生负载，逐步增大 batch 至显存耗尽。以 FP16 基线为比较对象、显存上限相近时，KIVI 可容纳最高 4 倍 batch size，吞吐为基线的 2.35 到 3.47 倍。结果受论文所用模型、硬件和负载限制，不能直接外推为生产服务的速度保证。量化效果取决于方案、模型、任务和运行条件；长上下文质量也不能只凭位宽判断。",[14,95,96],{"id":96},"怎样判断服务是否改过精度",[10,98,99],{},"运营者采用低精度配置有成本动机：较小的权重与 cache 能释放显存，有机会容纳更多并发请求。但研究论文和开源引擎文档证明的是技术可行性，不是某家服务实际做过哪次变更。同样的体感还可能来自模型 revision 或路由、系统提示词、采样参数和推理预算的变化。",[10,101,102],{},"用户侧要判断服务是否变过，需要保留上线初期的同题输出作基线，固定可见的模型版本、输入、采样参数和工具设置，再重复测试短、长上下文任务并盲评。没有历史基线，只能测量当前表现；如果系统提示词、路由或模型 revision 不公开，前后结果的变化也无法定位到量化。",[10,104,105,106,109,110,113],{},"能访问 serving 配置时，可以固定 checkpoint ",[22,107,108],{},"revision","、代码与量化内核、硬件、tokenizer、模板和采样配置，分别比较权重量化与 ",[22,111,112],{},"kv_cache_dtype","。两者可能相互作用，因此还可以补齐四种配置：都不量化、仅量化权重、仅量化 cache、两者都量化。若权重量化方案同时改变激活精度，测到的是整套方案的效果；要单独归因权重，还需固定激活精度或选用 weight-only 方案。",[10,115,116],{},"因此，量化是“体感变弱”的一种可检验解释。要断定某项服务上线后才切换了精度，还需要部署变更记录或能够隔离变量的服务端对照。",{"title":118,"searchDepth":119,"depth":120,"links":121},"",2,3,[122,123,124],{"id":16,"depth":119,"text":17},{"id":61,"depth":119,"text":62},{"id":96,"depth":119,"text":96},"技术杂谈","2026-09-29 21:44:08","权重与 KV cache 的量化都能降低 serving 成本，也可能改变输出；但体感变弱不足以证明服务后来切换了精度。",false,"md",{},true,"\u002Fposts\u002Fwhy-llm-seems-worse-after-launch",{"title":5,"description":127},"posts\u002Fwhy-llm-seems-worse-after-launch",[136,137,138],"LLMs","Serving","量化","faqWYnPlzyu93-KhjKazoPGQKav9S0OY5xeuEC7P1i8",[141,151,156,160,169,175,179,181,191,197,203,212,216,220,224,234,241,247,251,256,264,271,279,284,290,296,301,306,313,320,327,333,340,348,354,360,366,372,377,385],{"path":142,"title":143,"description":144,"date":145,"categories":146,"tags":147,"draft":128},"\u002Fposts\u002Fsurvival-welcome-to-university","生存手册——欢迎来到大学","大学同时生产三样东西：可验证技能、认证信号和驯化资本。你付了四年的钱，真正拿到手的是哪一样，取决于你毕业时要进的那个市场在为什么付费。","2026-10-12 20:00:00","生活杂谈",[148,149,150],"大学","求职","方法论",{"path":152,"title":153,"description":118,"date":154,"categories":118,"tags":155,"draft":128},"\u002Fposts\u002Fgp-tab-测试","gp-tab-测试","2026-10-12 10:51:22",null,{"path":157,"title":158,"description":118,"date":159,"categories":118,"tags":155,"draft":131},"\u002Fposts\u002Fnetwork-clean","怎么让网络变得干净","2026-10-11 19:40:02",{"path":161,"title":162,"description":163,"date":164,"categories":146,"tags":165,"draft":128},"\u002Fposts\u002Fthe-age-of-hesitation","[搬运] 不敢投入的时代","为什么我们现在越来越不敢投入了。很多时候我们以为的选择，比如你有两个工作offer，现在选择好像是我到底应该考公还是考编。明明这两个选项都不是真正的选项，因为你还没有开始，也不知道最后能不能成。这个时候我们就已经开始犹豫了。","2026-10-10 11:44:12",[166,167,168],"翻译","搬运","心理学",{"path":170,"title":171,"description":172,"date":173,"categories":146,"tags":174,"draft":128},"\u002Fposts\u002Fwhy-happiness-look-like-same","[搬运] 为什么大家的幸福都长一个样？","为什么大家的幸福都长一个样？一种认知被广泛传播后似乎变得不断扁平，这种感觉很奇怪。","2026-10-09 13:57:15",[166,167,168],{"path":176,"title":177,"description":118,"date":178,"categories":146,"tags":155,"draft":128},"\u002Fposts\u002Feverything-is-paid-twice","任何事情都要被消费两次","2026-10-06 12:30:06",{"path":132,"title":5,"description":127,"date":126,"categories":125,"tags":180,"draft":128},[136,137,138],{"path":182,"title":183,"description":184,"date":185,"categories":186,"tags":187,"draft":128},"\u002Fposts\u002Fspinning-up-policy-optimization","[译] 深度强化学习入门（三）：策略优化入门","OpenAI Spinning Up 教程第三部分的全文翻译：推导最简单的策略梯度、log-derivative trick、EGLP 引理、reward-to-go 与基线，配 PyTorch 示例代码。","2026-09-28 17:49:25","技术",[188,189,190,166],"强化学习","RL","Spinning Up",{"path":192,"title":193,"description":194,"date":195,"categories":186,"tags":196,"draft":128},"\u002Fposts\u002Fspinning-up-rl-intro-2","[译] 深度强化学习入门（二）：强化学习算法的分类","OpenAI Spinning Up 第二部分的全文翻译：无模型与基于模型的 RL、策略优化与 Q-learning 的取舍，以及规划、专家迭代和经验增强等方法。","2026-09-28 10:20:21",[188,189,190,166],{"path":198,"title":199,"description":200,"date":201,"categories":186,"tags":202,"draft":128},"\u002Fposts\u002Fspinning-up-rl-intro","[译] 深度强化学习入门（一）：RL 的关键概念","OpenAI Spinning Up 教程第一部分的全文翻译：智能体与环境、状态与观测、动作空间、策略、轨迹、奖励与回报、价值函数、贝尔曼方程、优势函数，附 MDP 形式化。","2026-09-28 09:29:40",[188,189,190,166],{"path":204,"title":205,"description":206,"date":207,"categories":146,"tags":208,"draft":131},"\u002Fposts\u002Fwhen-machines-come-alive","冰箱之后，生活还会怎样改变？","从厨房里的果蝇想到冰箱、移动互联网与 AI 的多轮调用：AGI 已经到来，只是昂贵的智能会怎样改变生活？","2026-09-25 06:36:22",[209,210,211],"AI","随笔","科技",{"path":213,"title":214,"description":118,"date":215,"categories":118,"tags":155,"draft":131},"\u002Fposts\u002Ftest","test","2026-09-20 06:37:44",{"path":217,"title":218,"description":118,"date":219,"categories":118,"tags":155,"draft":131},"\u002Fposts\u002Fmathematical-foundations-of-llms-1","LLM的数学基础（1）：","2026-09-19 22:59:55",{"path":221,"title":222,"description":118,"date":223,"categories":118,"tags":155,"draft":131},"\u002Fposts\u002Ftowards-maximize-serving-capatity","如何在 offline bench 里最大化 LLM serving system 的吞吐","2026-09-19 05:45:05",{"path":225,"title":226,"description":227,"date":228,"categories":125,"tags":229,"draft":128},"\u002Fposts\u002Fllm-memory-compute-tradeoff","LLM 的 Memory-Compute Trade-off：从 MHA 到 MLA、Sparse 与 Linear Attention","ds4.1f 上线了。它的架构 novel 在哪，便宜和快又是从哪省出来的。","2026-09-11 12:00:00",[136,230,231,232,233],"Attention","MLA","DeepSeek","长上下文",{"path":235,"title":236,"description":237,"date":238,"categories":125,"tags":239,"draft":131},"\u002Fposts\u002Fmarkdown-extensions","Markdown 扩展语法速查","本站 Markdown 支持的扩展语法演示与速查：admonition 提示块、GitHub 仓库卡片、spoiler 遮罩、ABC 乐谱、GP 乐谱、音频播放器、浏览器可运行代码块。","2026-09-01 15:30:00",[240],"博客",{"path":242,"title":243,"description":244,"date":245,"categories":146,"tags":246,"draft":128},"\u002Fposts\u002Fwhy-we-still-need-to-write-a-blog","为什么还需要手写博客？","AI这么强，还要古法写博客吗？","2026-09-01 13:17:53",[240],{"path":248,"title":249,"description":118,"date":250,"categories":118,"tags":155,"draft":131},"\u002Fposts\u002Fllm-serving-1","LLM Serving（一）:以在H200上部署GLM-5.2为例","2026-07-21 06:15:44",{"path":252,"title":253,"description":118,"date":254,"categories":255,"tags":155,"draft":128},"\u002Fposts\u002Finteresting-music-rhythm","有趣的音乐节奏","2026-07-16 12:47:24","音乐",{"path":257,"title":258,"description":259,"date":260,"categories":186,"tags":261,"draft":128},"\u002Fposts\u002Ftowards-monosemanticity","LLM 可解释性（一）：SAE与单语义","读 Anthropic《Towards Monosemanticity》。从难以解释的神经元出发, 讲清superposition为什么让单个神经元混成一团, SAE怎么把多语义的激活分解成单语义特征。","2026-07-12 22:00:00",[136,262,263],"可解释性","SAE",{"path":265,"title":266,"description":267,"date":268,"categories":146,"tags":269,"draft":131},"\u002Fposts\u002Fpiketty-ai-china-k-shaped","用皮凯蒂的镜头看 AI 与中国 K 型社会","一次凌晨的对话整理。从 r>g 出发，看 AI 时代的资本分配、中国的世袭化趋势、体制内的二元劳动力市场，以及一个普通 00 后在一线打拼的真实牌面。","2026-06-24 03:00:00",[210,270,209],"经济",{"path":272,"title":273,"description":118,"date":274,"categories":118,"tags":275,"draft":131},"\u002Fposts\u002Fclaude-code-compact-behavior","cluade code compact 行为","2026-06-23 20:17:42",[276,277,278],"Agent","AI Agent","Claude Code",{"path":280,"title":281,"description":118,"date":282,"categories":186,"tags":283,"draft":128},"\u002Fposts\u002Fclaude-code-settings","Claude Code 配置分享","2026-06-23 14:22:12",[136,277,276,278],{"path":285,"title":286,"description":287,"date":288,"categories":125,"tags":289,"draft":128},"\u002Fposts\u002Fmy-thoughts-on-the-ai-industry","简单聊聊对 AI Agent 行业的理解","写这篇文章的初衷，是我发觉逐渐忘了前两个月的 AI\u002FLLMs 是什么样的能力和情况了。意识中似乎某些在突然加速，但理性和直觉告诉我这不对，作为行内人，我知道这是一个渐进的、累积的过程。我想自己回忆一下，Agent的发展。","2026-06-20 13:14:25",[136,277,276],{"path":291,"title":292,"description":118,"date":293,"categories":186,"tags":294,"draft":128},"\u002Fposts\u002Frotational-positional-encoding","RoPE","2026-06-17 18:31:40",[136,295,292],"Transformer",{"path":297,"title":298,"description":118,"date":299,"categories":186,"tags":300,"draft":128},"\u002Fposts\u002Fintroduction-to-large-language-model","大语言模型入门基础","2026-06-16 22:36:11",[295,136],{"path":302,"title":303,"description":118,"date":304,"categories":125,"tags":305,"draft":128},"\u002Fposts\u002Fthought-about-coding-agent","关于 Coding Agent 使用的遐思","2026-06-03 17:54:20",[276],{"path":307,"title":308,"description":309,"date":310,"categories":186,"tags":311,"draft":128},"\u002Fposts\u002Fabout-ssh-usage","关于SSH的使用——客户端篇","介绍SSH客户端的用法。","2025-06-17 02:44:57",[312],"工具",{"path":314,"title":315,"description":316,"date":317,"categories":125,"tags":318,"draft":128},"\u002Fposts\u002Fopensourcesoftwarestarttutorial","如何开始一个开源项目？","浅谈我对开源项目的认知。","2025-04-17 18:00:00",[319],"开源",{"path":321,"title":322,"description":323,"date":324,"categories":325,"tags":326,"draft":128},"\u002Fposts\u002Fmotivation","渴求动力","有意识的生活并主动地行动会使得生活更美好。","2025-03-05 00:44:57","生活-情感",[210],{"path":328,"title":329,"description":330,"date":331,"categories":325,"tags":332,"draft":128},"\u002Fposts\u002F2024summary","2024年终总结","啊，痛彻的时光；啊，悲怆的人！","2025-01-01 01:00:00",[210],{"path":334,"title":335,"description":336,"date":337,"categories":186,"tags":338,"draft":128},"\u002Fposts\u002Fvps-crash-startup","快速搭建个人VPS","如何快速搭建个人VPS。","2024-12-31 19:00:00",[339],"VPS",{"path":341,"title":342,"description":343,"date":344,"categories":186,"tags":345,"draft":128},"\u002Fposts\u002Fdata-label-software","AI数据（自动）标注软件概述","对市面上目前的AI数据（自动）标注软件进行调研。","2024-12-20 14:00:00",[346,347],"数据标注","深度学习",{"path":349,"title":350,"description":351,"date":352,"categories":325,"tags":353,"draft":128},"\u002Fposts\u002Fself-destruction","生活的崩坏与自我毁灭","是什么在牵引着我？我的行为，我的思想？我的感受？我所处的地方？它们在何处？","2024-12-19 12:00:00",[210],{"path":355,"title":356,"description":357,"date":358,"categories":186,"tags":359,"draft":128},"\u002Fposts\u002Fcs194-196_llmagents","LLM Agents 课程笔记","UCB CS294\u002F194-196 Large Language Model Agents 的课程笔记。","2024-11-30 20:00:00",[276],{"path":361,"title":362,"description":363,"date":364,"categories":325,"tags":365,"draft":128},"\u002Fposts\u002Fwhatsyourworth","价值、位置","你们的生存究竟有何价值？如果毫无价值，你们究竟为何存在？","2024-11-29 02:00:00",[210],{"path":367,"title":368,"description":369,"date":370,"categories":325,"tags":371,"draft":128},"\u002Fposts\u002Fblogrelifever2","重启博客v2","忙里偷闲学习前端，还是决定结合想法手搓一个自己的博客。","2024-11-25 00:00:00",[240],{"path":373,"title":374,"description":375,"date":376,"categories":325,"tags":155,"draft":128},"\u002Fposts\u002Fblogrelife","重启博客","在我的表达欲与现实所需(问题解决记录)驱动下，我决定重写博客。","2024-10-21 20:44:57",{"path":378,"title":379,"description":380,"date":381,"categories":325,"tags":382,"draft":128},"\u002Fposts\u002Frevival","240205 复活","好吧，断更了许久，我又终于回来了，我又终于回来了。","2024-02-05 20:00:00",[210,383,384],"生活","返乡",{"path":386,"title":387,"description":388,"date":389,"categories":325,"tags":390,"draft":128},"\u002Fposts\u002Fclouds","「230717」云","我已经很久忘记了不我已经很少把自己的想法在lofter上发布的。","2023-07-17 20:00:00",[210,391],"散文",1791871062123]