[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"sanity-nsgTpVaepwam-mqKovw4Qn-UT_pV0rwZH2Nxf3F1tVI":3,"sanity-4Q7syukK2g9vpiga0B0MbyzLWYm0y0iK-xTV159YNws":358},{"data":4,"sourceMap":-1},{"latestPodcast":5,"latestReleases":14,"post":39,"recent":333},[6],{"_id":7,"publishedAt":8,"slug":9,"sponsored":12,"title":13},"d52b334e-4233-4e7d-a10b-bfba1472c2fb","2026-07-07T07:40:00.000Z",{"_type":10,"current":11},"slug","agent-orchestration-is-so-two-years-ago",null," Agent orchestration is so two-years ago",[15,21,27,33],{"_id":16,"publishedAt":17,"slug":18,"title":20},"eb5b66eb-9410-4329-83bb-22bbff39402a","2026-04-28T13:00:00.000Z",{"_type":10,"current":19},"turn-scattered-knowledge-into-trusted-intelligence","Turning scattered knowledge into trusted intelligence: Stack Internal 2026.3",{"_id":22,"publishedAt":23,"slug":24,"title":26},"369c2401-b62e-4a37-8ff8-bf603023ecad","2026-03-02T15:03:00.988Z",{"_type":10,"current":25},"what-s-new-at-stack-overflow-march-2026","What’s new at Stack Overflow: March 2026",{"_id":28,"publishedAt":29,"slug":30,"title":32},"5e9053a4-07ea-447c-91ea-29e0b6228537","2026-02-02T15:00:00.000Z",{"_type":10,"current":31},"what-s-new-at-stack-overflow-february-2026","What’s new at Stack Overflow: February 2026",{"_id":34,"publishedAt":35,"slug":36,"title":38},"a1b538eb-a8a6-46d0-80a1-ac70ec9bb935","2026-01-05T10:00:00.000-05:00",{"_type":10,"current":37},"what-s-new-at-stack-overflow-january-2026","What’s new at Stack Overflow: January 2026",{"_createdAt":40,"_id":41,"_rev":42,"_system":43,"_type":46,"_updatedAt":47,"author":48,"body":65,"comments":299,"dateUrl":300,"excerpt":301,"image":302,"product":12,"publishedAt":306,"slug":307,"sponsored":12,"tags":309,"title":332,"visible":299},"2025-02-14T22:40:06Z","ec74f3ce-0e51-4de2-9492-e6ee736801ed","OuLxq6F1ZwNJHk6WTQWZX2",{"base":44},{"id":41,"rev":45},"vn3UzGZJyacwgllS8WaxOs","blogPost","2026-05-11T17:04:47Z",[49],{"_createdAt":50,"_id":51,"_rev":52,"_type":53,"_updatedAt":50,"avatar":54,"bio":59,"employee":60,"name":61,"role":62,"slug":63},"2024-04-04T14:34:48Z","12169ed9-78d5-4f82-adb4-e956348db389","l85YFb2NNiT2L5SEvOgP0n","blogAuthor",{"_type":55,"asset":56},"image",{"_ref":57,"_type":58},"image-7c75ba835850e53201ea979015e8b8f449e126da-400x400-png","reference","I’m currently the Director of AI at Rebuy, a personalized search and recommendations platform for D2C e-commerce brands. Prior to Rebuy, I was a Research Scientist at Alegion. Additionally, I worked for Salesforce Commerce Cloud for two years. ","none","Cameron R. Wolfe, PhD","Director of AI at Rebuy",{"_type":10,"current":64},"cameron-r-wolfe-phd",[66,77,85,93,100,108,115,123,130,138,145,153,160,170,178,185,193,200,209,217,225,233,240,248,255,263,271,279,287,295],{"_key":67,"_type":68,"children":69,"markDefs":75,"style":76},"a69161d2f8ad","block",[70],{"_key":71,"_type":72,"marks":73,"text":74},"48f1bae80c8a0","span",[],"There are many variants of LoRA you can use to train a specialized LLM on your own data. Here’s an overview of all (or at least most) of the techniques that are out there.",[],"normal",{"_key":78,"_type":68,"children":79,"markDefs":84,"style":76},"d3947b206203",[80],{"_key":81,"_type":72,"marks":82,"text":83},"e578d1d5397c0",[],"",[],{"_key":86,"_type":68,"children":87,"markDefs":92,"style":76},"05755a5d50d3",[88],{"_key":89,"_type":72,"marks":90,"text":91},"bbfc97613f150",[],"LoRA models the update derived for a model’s weights during finetuning with a low rank decomposition, implemented in practice as a pair of linear projections. LoRA leaves the pretrained layers of the LLM fixed and injects a trainable rank decomposition matrix into each layer of the model.",[],{"_key":94,"_type":68,"children":95,"markDefs":99,"style":76},"382c1954a963",[96],{"_key":97,"_type":72,"marks":98,"text":83},"139f1561a67d0",[],[],{"_key":101,"_type":68,"children":102,"markDefs":107,"style":76},"c1d5f2361250",[103],{"_key":104,"_type":72,"marks":105,"text":106},"264eeba323370",[],"QLoRA is (arguably) the most popular LoRA variant and uses model quantization techniques to reduce memory usage during finetuning while maintaining (roughly) equal levels of performance. QLoRA uses 4-bit quantization on the pretrained model weights and trains LoRA modules on top of this. In practice, QLoRA saves memory at the cost of slightly-reduced training speed.",[],{"_key":109,"_type":68,"children":110,"markDefs":114,"style":76},"cff395349a11",[111],{"_key":112,"_type":72,"marks":113,"text":83},"ea29f838eb890",[],[],{"_key":116,"_type":68,"children":117,"markDefs":122,"style":76},"8ff688911d73",[118],{"_key":119,"_type":72,"marks":120,"text":121},"ca5ef891c8c70",[],"QA-LoRA is an extension of LoRA\u002FQLoRA that further reduces the computational burden of training and deploying LLMs. It does this by combining parameter-efficient finetuning with quantization (i.e., group-wise quantization applied during training\u002Finference).",[],{"_key":124,"_type":68,"children":125,"markDefs":129,"style":76},"d590e1c81b88",[126],{"_key":127,"_type":72,"marks":128,"text":83},"a66c16f2a2df0",[],[],{"_key":131,"_type":68,"children":132,"markDefs":137,"style":76},"539e8deb96fe",[133],{"_key":134,"_type":72,"marks":135,"text":136},"d170fb3eb71f0",[],"LoftQ studies a similar idea to QA-LoRA—applying quantization and LoRA finetuning on a pretrained model simultaneously.",[],{"_key":139,"_type":68,"children":140,"markDefs":144,"style":76},"152ebf03647e",[141],{"_key":142,"_type":72,"marks":143,"text":83},"c567645aab000",[],[],{"_key":146,"_type":68,"children":147,"markDefs":152,"style":76},"66745b0876a4",[148],{"_key":149,"_type":72,"marks":150,"text":151},"f70a56a23aa10",[],"LongLoRA attempts to cheaply adapt LLMs to longer context lengths using a parameter-efficient (LoRA-based) finetuning scheme. In particular, we start with a pretrained model and finetune it to have a longer context length. This finetuning is made efficient by:",[],{"_key":154,"_type":68,"children":155,"markDefs":159,"style":76},"c82112ff778a",[156],{"_key":157,"_type":72,"marks":158,"text":83},"ab6e5805e4770",[],[],{"_key":161,"_type":68,"children":162,"level":167,"listItem":168,"markDefs":169,"style":76},"6412a102b242",[163],{"_key":164,"_type":72,"marks":165,"text":166},"f5b4545412fa0",[],"Using sparse local attention instead of dense global attention (optional at inference time).",1,"number",[],{"_key":171,"_type":68,"children":172,"level":167,"listItem":168,"markDefs":177,"style":76},"8952f365dcc9",[173],{"_key":174,"_type":72,"marks":175,"text":176},"f89554d1b1e00",[],"Using LoRA (authors find that this works well for context extension).",[],{"_key":179,"_type":68,"children":180,"markDefs":184,"style":76},"cff8a416e861",[181],{"_key":182,"_type":72,"marks":183,"text":83},"b01201dab0420",[],[],{"_key":186,"_type":68,"children":187,"markDefs":192,"style":76},"f45b477ccf4f",[188],{"_key":189,"_type":72,"marks":190,"text":191},"a6b3c61981a00",[],"S-LoRA aims to solve the problem of deploying multiple LoRA modules that are used to adapt the same pretrained model to a variety of different tasks. Put simply, S-LoRA does the following to serve thousands of LoRA modules on a single GPU (or across GPUs):",[],{"_key":194,"_type":68,"children":195,"markDefs":199,"style":76},"ca2003ca155d",[196],{"_key":197,"_type":72,"marks":198,"text":83},"66a53e0829b80",[],[],{"_key":201,"_type":68,"children":202,"level":167,"listItem":207,"markDefs":208,"style":76},"dc4f5cc260a0",[203],{"_key":204,"_type":72,"marks":205,"text":206},"69624e2b72fb0",[],"Stores all LoRA modules in main memory.","bullet",[],{"_key":210,"_type":68,"children":211,"level":167,"listItem":207,"markDefs":216,"style":76},"08e10fbc37a8",[212],{"_key":213,"_type":72,"marks":214,"text":215},"2de6b7af8b970",[],"Puts modules being used to run the current query into GPU memory.",[],{"_key":218,"_type":68,"children":219,"level":167,"listItem":207,"markDefs":224,"style":76},"7086401c0852",[220],{"_key":221,"_type":72,"marks":222,"text":223},"ffc2cd806fe70",[],"Uses unified paging to allocate GPU memory and avoid fragmentation.",[],{"_key":226,"_type":68,"children":227,"level":167,"listItem":207,"markDefs":232,"style":76},"303d9fd79864",[228],{"_key":229,"_type":72,"marks":230,"text":231},"888c8bfd2db70",[],"Proposes a new tensor parallelism strategy to batch LoRA computations.",[],{"_key":234,"_type":68,"children":235,"markDefs":239,"style":76},"0b3415bf34f3",[236],{"_key":237,"_type":72,"marks":238,"text":83},"95ea915b0dc10",[],[],{"_key":241,"_type":68,"children":242,"markDefs":247,"style":76},"a86dcf702fdf",[243],{"_key":244,"_type":72,"marks":245,"text":246},"fefed0d452d10",[],"Many other LoRA variants exist as well…",[],{"_key":249,"_type":68,"children":250,"markDefs":254,"style":76},"1ed8096eb0da",[251],{"_key":252,"_type":72,"marks":253,"text":83},"0b70e1794d880",[],[],{"_key":256,"_type":68,"children":257,"level":167,"listItem":207,"markDefs":262,"style":76},"9087c53ed561",[258],{"_key":259,"_type":72,"marks":260,"text":261},"a471d79ce0930",[],"LQ-LoRA: uses a more sophisticated quantization scheme within QLoRA that performs better and can be adapted to a target memory budget.",[],{"_key":264,"_type":68,"children":265,"level":167,"listItem":207,"markDefs":270,"style":76},"71bd47f5a295",[266],{"_key":267,"_type":72,"marks":268,"text":269},"48c772d019dc0",[],"MultiLoRA: extension of LoRA that better handles complex multi-task learning scenarios.",[],{"_key":272,"_type":68,"children":273,"level":167,"listItem":207,"markDefs":278,"style":76},"3af9adbf0e4c",[274],{"_key":275,"_type":72,"marks":276,"text":277},"571bd3dbfce30",[],"LoRA-FA: freezes half of the low-rank decomposition matrix (i.e., the A matrix within the product AB) to further reduce memory overhead.",[],{"_key":280,"_type":68,"children":281,"level":167,"listItem":207,"markDefs":286,"style":76},"197c135439c4",[282],{"_key":283,"_type":72,"marks":284,"text":285},"a3cbf422bb4a0",[],"Tied-LoRA: leverages weight tying to further improve the parameter efficiency of LoRA.",[],{"_key":288,"_type":68,"children":289,"level":167,"listItem":207,"markDefs":294,"style":76},"adf1fe522735",[290],{"_key":291,"_type":72,"marks":292,"text":293},"0825ac827fc80",[],"GLoRA: extends LoRA to adapt pretrained model weights and activations to each task in addition to an adapter for each layer.",[],{"_key":296,"_type":55,"asset":297,"markDefs":12},"900046695fda",{"_ref":298,"_type":58},"image-6e51e6782d8114b0ebf3018df478d46cbbd6b92a-2416x1356-jpg",true,"2025\u002F02\u002F26","Want to train a specialized LLM on your own data? The easiest way to do this is with low rank adaptation (LoRA), but many variants of LoRA exist. ",{"_type":55,"asset":303,"attribution":305},{"_ref":304,"_type":58},"image-f4473993106f8912780f3034210971f5dd8391e1-12000x6293-jpg","Alexandra Francis","2025-02-26T17:00:00.000Z",{"_type":10,"current":308},"variants-of-lora",[310,317,321,325],{"_createdAt":311,"_id":312,"_rev":313,"_type":314,"_updatedAt":311,"slug":315,"title":316},"2023-05-23T16:43:21Z","wp-tagcat-se-tech","9HpbCsT2tq0xwozQfkc4ih","blogTag",{"current":316},"se-tech",{"_createdAt":311,"_id":318,"_rev":313,"_type":314,"_updatedAt":311,"slug":319,"title":320},"wp-tagcat-se-stackoverflow",{"current":320},"se-stackoverflow",{"_createdAt":311,"_id":322,"_rev":313,"_type":314,"_updatedAt":311,"slug":323,"title":324},"wp-tagcat-llm",{"current":324},"llm",{"_createdAt":311,"_id":326,"_rev":327,"_type":314,"_updatedAt":328,"slug":329,"title":331},"wp-tagcat-ai","fpDTFQqIDjNJIbHDKPBGpV","2025-01-30T16:19:01Z",{"current":330},"ai","AI","Variants of LoRA",[334,340,346,352],{"_id":335,"publishedAt":336,"slug":337,"sponsored":12,"title":339},"76c9771b-34e6-4d98-8641-ecefc711f0ef","2026-07-06T15:23:34.559Z",{"_type":10,"current":338},"when-the-sensor-starts-thinking-snortml-agentic-ai-and-the-evolving-architecture-of-intrusion-detection","When the sensor starts thinking: SnortML, agentic AI, and the evolving architecture of intrusion detection",{"_id":341,"publishedAt":342,"slug":343,"sponsored":12,"title":345},"28e560af-f0aa-4d46-bd90-f435ad604aa7","2026-06-26T14:00:27.102Z",{"_type":10,"current":344},"paging-charity-how-can-engineering-leaders-avoid-becoming-bond-villains","Paging Charity! How can engineering leaders avoid becoming Bond villains?",{"_id":347,"publishedAt":348,"slug":349,"sponsored":12,"title":351},"4b22c2a3-3779-4966-93eb-5230391dbdce","2026-06-23T14:08:58.595Z",{"_type":10,"current":350},"your-ai-shipped-a-backend-that-boots-that-is-the-whole-problem","Your AI shipped a backend that boots. That is the whole problem.",{"_id":353,"publishedAt":354,"slug":355,"sponsored":12,"title":357},"5cf362e1-fe7b-45af-b69c-914731c6a052","2026-06-23T14:00:00.000Z",{"_type":10,"current":356},"the-2026-developer-survey-is-now-open-for-human-developers-only","The 2026 Developer Survey is now open (for human developers only)!",{"data":359,"sourceMap":-1},{"count":360,"lastTimestamp":12},0]