[et_pb_section fb_built=”1″ _builder_version=”4.16″ _module_preset=”default” custom_margin=”0px||||false|false” custom_padding=”0px||||false|false” locked=”off” global_colors_info=”{}”][et_pb_row _builder_version=”4.16″ _module_preset=”default” width=”100%” max_width=”100%” custom_margin=”||||false|false” custom_padding=”0px||||false|false” global_colors_info=”{}”][et_pb_column type=”4_4″ _builder_version=”4.16″ _module_preset=”default” global_colors_info=”{}”][et_pb_text _builder_version=”4.27.9″ _module_preset=”default” ol_line_height=”1.7em” header_2_line_height=”1.7em” header_3_line_height=”1.7em” custom_margin=”||-10px||false|false” custom_padding=”||0px||false|false” hover_enabled=”0″ locked=”off” global_colors_info=”{}” sticky_enabled=”0″]<\/p>\n
\nSmaller models can do far more than they could a year ago. But larger models can still raise the capability ceiling, and the workloads surrounding them require more memory than parameter count alone suggests.<\/p>\n<\/blockquote>\n
\u00a0<\/span>\u00a0<\/span><\/p>\n
Local AI is improving quickly. Better training, distillation, quantization, and model architectures are allowing smaller models to handle work that recently required much larger ones. Models designed for PCs and workstations can now reason, understand images, call tools, work with documents, and run as part of an agentic workflow.\u00a0<\/span>\u00a0<\/span><\/p>\n
That is changing what can be done locally. It doesn\u2019t mean model size, or memory capacity, has stopped mattering.\u00a0<\/span>\u00a0<\/span><\/p>\n
Smaller models are becoming more capable, but larger models can still provide better reasoning, accuracy, knowledge coverage, multilingual performance, and reliability on difficult tasks. Meanwhile, the definition of a useful local AI workload is expanding beyond loading one model and answering one prompt. We now expect local AI to process larger contexts, understand multiple types of data, use tools, retrieve private information, retain previous work, and sometimes run several models or agents at once.\u00a0<\/span>\u00a0<\/span><\/p>\n
The model may be more efficient, but the complete workload is not necessarily smaller.<\/span>\u00a0<\/span><\/p>\n
\u00a0<\/span>\u00a0<\/span><\/p>\n
Smaller models raise the floor, not the ceiling<\/h3>\n
Parameter count is an imperfect measure of model capability. A well-trained 12B model may outperform an older model several times its size. Quantization can substantially reduce the memory required to run a model. Mixture-of-Experts (MoE) architectures can activate only a small portion of a model\u2019s parameters for each token, reducing compute requirements.\u00a0<\/span>\u00a0<\/span><\/p>\n
These advances make useful local AI possible on a wider range of systems. But they don\u2019t make larger models irrelevant.\u00a0<\/span>\u00a0<\/span><\/p>\n
Model quality depends on architecture, training data, post-training, precision, and the workload being evaluated, not parameter count alone. Still, difficult reasoning, coding, multilingual, and multimodal tasks can benefit from the additional capacity of a larger model. A smaller model may perform well on a benchmark or narrow task without providing the same consistency across a complex enterprise workflow.\u00a0<\/span>\u00a0<\/span><\/p>\n
The practical question goes beyond whether small or large models have \u201cwon.\u201d It\u2019s more about which model provides acceptable quality for the workload, and whether the local system has enough memory to run it.\u00a0<\/span>\u00a0<\/span><\/p>\n
\u00a0<\/span>\u00a0<\/span><\/p>\n
Capability adds memory beyond the model<\/h3>\n
Model weights are only the most obvious part of the memory requirement. A modern local-AI application may also need memory for:\u00a0<\/span>\u00a0<\/span><\/p>\n
\u00a0<\/span>\u00a0<\/span><\/p>\n
\n
- Long context and its associated KV cache\u00a0<\/span>\u00a0<\/span><\/li>\n<\/ul>\n
\n
- Retrieved documents and enterprise data\u00a0<\/span>\u00a0<\/span><\/li>\n<\/ul>\n
\n
- Image, audio, or video inputs\u00a0<\/span>\u00a0<\/span><\/li>\n<\/ul>\n
\n
- Tool definitions, calls, and results\u00a0<\/span>\u00a0<\/span><\/li>\n<\/ul>\n
\n
- Agent plans and previous actions\u00a0<\/span>\u00a0<\/span><\/li>\n<\/ul>\n
\n
- Embedding and reranking models\u00a0<\/span>\u00a0<\/span><\/li>\n<\/ul>\n
\n
- Multiple users, sessions, or agents\u00a0<\/span>\u00a0<\/span><\/li>\n<\/ul>\n
\n
- Reusable prompts and precomputed context\u00a0<\/span>\u00a0<\/span><\/li>\n<\/ul>\n
\u00a0<\/span>\u00a0<\/span><\/p>\n
These requirements accumulate while the application runs.\u00a0<\/span>\u00a0<\/span><\/p>\n
A model that fits when the application starts may leave too little capacity for a long context window. An agent may begin with a manageable prompt and then add source documents, search results, tool output, code, plans, and prior responses over dozens of steps.\u00a0<\/span>\u00a0<\/span><\/p>\n
Concurrency makes the problem worse. Each active session may require its own KV cache. Several agents working on different parts of a task can multiply the memory footprint even when they share the same base model. Fitting the model is necessary, but it\u2019s\u00a0 no longer sufficient.\u00a0<\/span>\u00a0<\/span><\/p>\n
\u00a0<\/span>\u00a0<\/span><\/p>\n
Long context is not free<\/h3>\n
Long context has become a standard model feature. Current models advertise context windows of hundreds of thousands or even a million tokens. That doesn\u2019t mean every local system can use the entire window effectively.\u00a0<\/span>\u00a0<\/span><\/p>\n
During inference, the system creates a KV cache containing information derived from previously processed tokens. The KV cache allows the model to continue without recomputing its entire history for every new token. As context grows, so does the KV cache. Its size also increases with model architecture, batch size, and the number of concurrent sessions.\u00a0<\/span>\u00a0<\/span><\/p>\n
An application can control memory consumption by dropping older context, summarizing it, or recomputing information when needed. Each option has a cost. The model may lose useful detail, spend additional compute reconstructing previous work, or take longer to respond.\u00a0<\/span>\u00a0<\/span><\/p>\n
A large advertised context window is therefore both a model capability and an infrastructure requirement.\u00a0<\/span>\u00a0<\/span><\/p>\n
\u00a0<\/span>\u00a0<\/span><\/p>\n
Tool use turns a prompt into a workload<\/h3>\n
Tool calling itself is not especially memory-intensive. But what surrounds it can be. An agent begins with instructions and a task. It may then inspect files, call an API, search a database, run code, evaluate the result, revise its plan, and try again. Relevant outputs are added to the agent\u2019s working context so it can decide what to do next.\u00a0<\/span>\u00a0<\/span><\/p>\n
A coding agent may repeatedly work across a large repository. A research agent may process many documents and retain citations, notes, and intermediate conclusions. A business agent may pull data from several systems before generating an answer.\u00a0<\/span>\u00a0<\/span><\/p>\n
The result is not one large prompt. It is an expanding sequence of prompts, retrieved information, tool results, and decisions. This is why agentic AI shifts the memory discussion from model capacity to workload capacity.\u00a0<\/span>\u00a0<\/span><\/p>\n
\u00a0<\/span>\u00a0<\/span><\/p>\n
Multimodal means more than adding an image<\/h3>\n
Local models are also becoming natively multimodal. A single application may process text, images, audio, documents, screen captures, or video. Some multimodal models handle these inputs within one architecture. Other applications use several specialized models connected in a pipeline. Either approach adds memory pressure.\u00a0<\/span>\u00a0<\/span><\/p>\n
The system may need to hold the primary language model, modality-specific components, intermediate representations, and the context created from the input. A local assistant that can inspect a document, understand its charts, listen to a meeting, and call business tools is more useful than a text-only chatbot.\u00a0<\/span>\u00a0<\/span><\/p>\n
It is also a larger workload.\u00a0<\/span>\u00a0<\/span><\/p>\n
\u00a0<\/span>\u00a0<\/span><\/p>\n
MoE models reduce compute, not total model capacity<\/h3>\n
MoE models provide another example of why model efficiency and memory demand can increase together. An MoE model contains multiple expert networks but activates only a subset for each token. This allows the model to provide greater total capacity without using every parameter for every computation. The active parameter count may be relatively small, while the complete expert set can still be very large.\u00a0<\/span>\u00a0<\/span><\/p>\n
Keeping every expert in GPU memory consumes capacity even though most are inactive at any given moment. Keeping fewer experts resident saves memory but requires the system to load an expert when the model routes a token to it.\u00a0<\/span>\u00a0<\/span><\/p>\n
MoE makes computation more efficient. It doesn\u2019t make the inactive experts disappear.\u00a0<\/span>\u00a0<\/span><\/p>\n
\u00a0<\/span>\u00a0<\/span><\/p>\n
Memory is becoming a hierarchy<\/h3>\n
The conventional response to a memory limit is to add more GPU memory or system DRAM. That remains the best answer when the entire workload must run at maximum speed. However, it isn\u2019t always practical.\u00a0<\/span>\u00a0<\/span><\/p>\n
GPU memory is expensive and limited. Workstation DRAM provides more capacity but is still finite. Flash provides substantially more capacity at lower cost, with higher latency than either.\u00a0<\/span>\u00a0<\/span><\/p>\n
The useful question is not whether flash is as fast as GPU memory\u2014it isn\u2019t. The real question now is whether every part of the workload needs to occupy the fastest memory tier all the time.\u00a0<\/span>\u00a0<\/span><\/p>\n
Active model data and current context need rapid access. Inactive MoE experts, older KV cache, reusable prompt prefixes, and precomputed RAG content may be able to reside in a larger memory tier until needed. Managing those resources intelligently can allow a local system to run a larger or longer workload than its GPU and DRAM capacity would otherwise support.\u00a0<\/span>\u00a0<\/span><\/p>\n
\u00a0<\/span>\u00a0<\/span><\/p>\n
Where Pascari aiDAPTIV can help<\/h3>\n
Pascari aiDAPTIV extends effective AI memory across GPU memory, system memory, and high-endurance flash. It integrates with supported AI runtimes so the software can manage specific AI data rather than treating the SSD as generic system storage. Depending on the workload and runtime, this can include model data, MoE experts, KV cache, or precomputed KV entries for frequently used RAG content.\u00a0<\/span>\u00a0<\/span><\/p>\n
That enables several practical outcomes:\u00a0<\/span>\u00a0<\/span><\/p>\n
\u00a0<\/span>\u00a0<\/span><\/p>\n
\n
- Run models whose complete weights exceed available GPU and system memory\u00a0<\/span>\u00a0<\/span><\/li>\n<\/ul>\n
\n
- Load MoE experts dynamically instead of keeping every expert resident\u00a0<\/span>\u00a0<\/span><\/li>\n<\/ul>\n
\n
- Retain more KV cache as sessions grow\u00a0<\/span>\u00a0<\/span><\/li>\n<\/ul>\n
\n
- Reuse previously computed context when stable shared context exists\u00a0<\/span>\u00a0<\/span><\/li>\n<\/ul>\n
\n
- Pre-cache frequently retrieved content to reduce repeated prefill\u00a0<\/span>\u00a0<\/span><\/li>\n<\/ul>\n
\n
- Run memory-constrained fine-tuning workloads without adding GPUs\u00a0<\/span>\u00a0<\/span><\/li>\n<\/ul>\n
<\/p>\n
The result is not identical to running the entire workload in GPU memory. Performance depends on how much data moves between tiers and how often the workload needs it. But when the alternatives are reducing the model, shortening the context, limiting concurrency, adding GPUs, or not running the workload at all, that trade can be useful.\u00a0<\/span>\u00a0<\/span><\/p>\n
\u00a0<\/span>\u00a0<\/span><\/p>\n
Local AI is expanding, not shrinking<\/h3>\n
Smaller models will continue to improve, and quantization and memory compression will improve with them. These advances will allow increasingly capable AI to run on PCs and workstations.\u00a0<\/span>\u00a0<\/span><\/p>\n
Developers will use that efficiency to do more. They will choose larger models when the workload benefits from them. They will use longer context, add multimodal input, connect more tools, retain more state, and run more workloads concurrently.\u00a0<\/span>\u00a0<\/span><\/p>\n
Memory efficiency expands what local AI can do. It does not end the need for memory. The relevant sizing question is no longer simply, \u201cHow large is the model?\u201d Today, the question is:\u00a0<\/span>\u00a0<\/span><\/p>\n
What does the complete workload need to do, and how much memory will it require once it starts doing it?\u00a0<\/span><\/i>\u00a0<\/span><\/p>\n
<\/p>\n
Learn more about <\/span><\/i>Pascari aiDAPTIV<\/span><\/i><\/a> and flash-accelereated memory extension for local AI workloads, or\u00a0 <\/span><\/i>contact a Pascari sales representative<\/span><\/i><\/a> today.\u00a0\u00a0\u00a0<\/span><\/i>\u00a0<\/span><\/p>\n
[\/et_pb_text][\/et_pb_column][\/et_pb_row][et_pb_row disabled_on=”on|on|on” _builder_version=”4.27.6″ _module_preset=”default” width=”100%” max_width=”100%” custom_margin=”||||false|false” custom_padding=”0px||||false|false” disabled=”on” saved_tabs=”all” locked=”off” global_colors_info=”{}”][et_pb_column type=”4_4″ _builder_version=”4.16″ _module_preset=”default” global_colors_info=”{}”][et_pb_text _builder_version=”4.27.4″ _module_preset=”default” global_colors_info=”{}”]<\/p>\n
Frequently Asked Questions (FAQ) :<\/strong><\/h3>\n
[\/et_pb_text][et_pb_toggle title=”What is an open-weight AI model?” _builder_version=”4.27.7″ _module_preset=”default” global_colors_info=”{}”]<\/p>\n
An open-weight AI model is a trained model whose learned parameters, or weights, are available for users to download, enabling organizations to deploy and potentially customize the model on infrastructure they control. Depending on licensing terms, users may also fine-tune the model with their own data. Open weights provide greater deployment choice without necessarily exposing the model\u2019s complete training data, source code, or development process.<\/p>\n
[\/et_pb_toggle][et_pb_toggle title=”Why are open-weight AI models important for private and local AI?” _builder_version=”4.27.7″ _module_preset=”default” global_colors_info=”{}”]<\/p>\n
Open-weight AI models enable organizations to run AI on infrastructure they control, reducing the need to send proprietary prompts, documents, and datasets to an external AI service. Local deployment can strengthen data sovereignty, privacy, model customization, and provider independence while giving organizations greater control over their AI applications. Open access still requires appropriate model evaluation, security controls, and governance.<\/p>\n
[\/et_pb_toggle][et_pb_toggle title=”Why do large AI models require so much memory during inference?” _builder_version=”4.27.7″ _module_preset=”default” global_colors_info=”{}”]<\/p>\n
Large AI models consume memory for model weights, application data, temporary working memory, and the KV cache that retains context during inference, so total memory demand can substantially exceed the size of the model weights alone. MoE architectures add another consideration because systems must store and access multiple specialized experts, even when each token activates only a subset.<\/p>\n
[\/et_pb_toggle][et_pb_toggle title=”Should organizations use a smaller AI model or add more infrastructure capacity?” _builder_version=”4.27.7″ _module_preset=”default” global_colors_info=”{}”]<\/p>\n
Organizations should choose between a smaller model and additional infrastructure capacity based on workload requirements for model capability, context length, concurrency, performance, cost, and deployment control. Smaller or more aggressively quantized models reduce memory demand but may require compromises. Expanding infrastructure can preserve access to larger models and workloads, but conventional GPU and system memory can increase hardware costs significantly.<\/p>\n
[\/et_pb_toggle][et_pb_toggle title=”Is local AI better than cloud AI for running open-weight models?” _builder_version=”4.27.7″ _module_preset=”default” global_colors_info=”{}”]<\/p>\n
Local AI provides greater control over models and data, while cloud AI can provide access to infrastructure that organizations may not want to purchase or operate themselves, so the appropriate architecture depends on workload and business requirements. Open-weight models make local deployment practical, but the article does not position local infrastructure as a universal replacement for cloud or frontier-scale systems.<\/p>\n
[\/et_pb_toggle][et_pb_toggle title=”How does Pascari aiDAPTIV\u2122 help organizations run larger AI models locally?” _builder_version=”4.27.7″ _module_preset=”default” global_colors_info=”{}”]<\/p>\n
Pascari aiDAPTIV\u2122 extends effective AI memory capacity by adding a flash tier alongside GPU memory and system memory, allowing local systems to run AI workloads that would otherwise exceed available GPU memory and DRAM. aiDAPTIV Cache Memory and aiDAPTIV Memory Management Middleware manage data across these tiers, keeping frequently needed data closer to compute while moving less-active data to flash.<\/p>\n
[\/et_pb_toggle][et_pb_toggle title=”How does aiDAPTIV support Mixture-of-Experts AI models?” _builder_version=”4.27.7″ _module_preset=”default” global_colors_info=”{}”]<\/p>\n
aiDAPTIV can support Mixture-of-Experts models by keeping frequently selected experts in GPU or system memory while loading other experts from flash as needed, increasing the effective capacity available for larger MoE deployments. This tiered approach does not make flash equivalent to DRAM or GPU memory, but it can enable models that would otherwise exceed conventional memory limits.<\/p>\n
[\/et_pb_toggle][et_pb_toggle title=”Can aiDAPTIV help AI systems support longer context windows and more concurrent users?” _builder_version=”4.27.7″ _module_preset=”default” global_colors_info=”{}”]<\/p>\n
aiDAPTIV can extend the capacity available for KV cache with flash, helping AI systems retain more context and support more concurrent activity when GPU memory and system memory become capacity constraints. This approach targets long-running conversations and agentic workflows, where growing KV cache requirements can place significant pressure on available memory during inference.<\/p>\n
[\/et_pb_toggle][et_pb_toggle title=”Does using flash as AI memory reduce inference performance?” _builder_version=”4.27.7″ _module_preset=”default” global_colors_info=”{}”]<\/p>\n
Using flash as an additional AI memory tier can introduce a performance tradeoff because flash has higher latency than DRAM or GPU memory, but it can make otherwise memory-constrained workloads deployable. Phison positions aiDAPTIV around this capacity-performance tradeoff: some workloads may run more slowly than on systems with enough high-speed memory to hold everything, while gaining the ability to run on more practical local infrastructure.<\/p>\n
[\/et_pb_toggle][et_pb_toggle title=”How can Phison help make open-weight AI infrastructure more accessible?” _builder_version=”4.27.7″ _module_preset=”default” global_colors_info=”{}”]<\/p>\n
Phison addresses a key open-weight AI infrastructure constraint through Pascari aiDAPTIV\u2122, which extends effective memory capacity with flash so organizations can run larger and more demanding workloads without relying exclusively on frontier-scale GPU infrastructure. The architecture expands deployment flexibility across notebooks, workstations, edge systems, and on-premises servers while helping organizations align infrastructure spending with workload requirements.<\/p>\n
[\/et_pb_toggle][\/et_pb_column][\/et_pb_row][\/et_pb_section]<\/p>\n","protected":false},"excerpt":{"rendered":"
Smaller models can do far more than they could a year ago. But larger models can still raise the capability ceiling, and the workloads surrounding them require more memory than parameter count alone suggests. \u00a0\u00a0 Local AI is improving quickly. Better training, distillation, quantization, and model architectures are allowing smaller models to handle work that […]<\/p>\n","protected":false},"author":79,"featured_media":90156,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_et_pb_use_builder":"on","_et_pb_old_content":"","_et_gb_content_width":"","inline_featured_image":false,"footnotes":""},"categories":[120,23],"tags":[22],"class_list":["post-90152","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai","category-all-posts","tag-long-content"],"acf":[],"_links":{"self":[{"href":"https:\/\/phisonblog.com\/zh-tw\/wp-json\/wp\/v2\/posts\/90152","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/phisonblog.com\/zh-tw\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/phisonblog.com\/zh-tw\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/phisonblog.com\/zh-tw\/wp-json\/wp\/v2\/users\/79"}],"replies":[{"embeddable":true,"href":"https:\/\/phisonblog.com\/zh-tw\/wp-json\/wp\/v2\/comments?post=90152"}],"version-history":[{"count":5,"href":"https:\/\/phisonblog.com\/zh-tw\/wp-json\/wp\/v2\/posts\/90152\/revisions"}],"predecessor-version":[{"id":90160,"href":"https:\/\/phisonblog.com\/zh-tw\/wp-json\/wp\/v2\/posts\/90152\/revisions\/90160"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/phisonblog.com\/zh-tw\/wp-json\/wp\/v2\/media\/90156"}],"wp:attachment":[{"href":"https:\/\/phisonblog.com\/zh-tw\/wp-json\/wp\/v2\/media?parent=90152"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/phisonblog.com\/zh-tw\/wp-json\/wp\/v2\/categories?post=90152"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/phisonblog.com\/zh-tw\/wp-json\/wp\/v2\/tags?post=90152"}],"curies":[{"name":"\u53ef\u6fd5\u6027\u7c89\u5291","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}