DDN Infinia: Overcoming AI Inference Workload Performance Unpredictability

As I noted in my post on CoreWeave’s Forge offering we explored at Tech Field Day’s AI Field Day #9, the AI DevOps workflow patterns have just recently shifted towards a Mobius loop from initial LLM training, through inference executions via AI agents, and then feeding back the lessons learned from inference back to training in hopes of improving inference in the immediate future.

Just 18 months ago, AI performance optimization focused heavily at improving training operations because of the intense demands for long-running and expensive operations across multiple xPUs. Thanks to multiple advancements – new open weight models, cheaper and smaller LLMs, and optimized parallel file systems like DDN’s EXAScaler – training demands are now simpler to analyze, curate, and balance, and that’s lead to dramatic improvements in LLM training cycles.

With the onset of agentic AI, however, the secondary problem now becomes the primary one: IT shops now must focus on optimizing inference well beyond a slew of simple chatbot responses and shift to inference at scale. The team at DDN acknowledged this dramatic shift with a concise summary of the challenges IT shops face when shifting towards scalable inference operations.

That means enabling a plethora of AI agents communicating across multiple storage endpoints that comprise a modern enterprise’s multi-cloud / multi-platform set of datasources – everything from databases to JSON documents to static media files – to locate, capture, and retain data to answer inference queries – also known as prefill.

Put more simply: Poor inference operation performance is directly related to any bottlenecks in getting precisely the needed data depending on the operation’s required context onto the closest GPUs as rapidly as possible so it can be then decoded and returned to the requesting agent.

I’ve Seen This Prefill Thing Before.

As a former Oracle University instructor who taught everything from basic database administration to how to tweak maximum performance from an Exadata Database Machine, I immediately recognized the same issues DBAs and their developers have struggled with over the past decade: How do we keep the hottest data as close as possible to a database’s memory structures to prevent unnecessary reads from physical storage that’s often several orders of magnitude slower than direct memory access?

The solution we often deployed as DBAs was the elegant in-memory database option (IMDB) available in every Oracle database instance since its 12c release almost a decade ago. With IMDB enabled, the database used statistics from historical access patterns to predict which chunks of data should be ready to be cached within memory ahead of time to significantly improve read/write access.

A skilled DBA could also partition database objects so that the most often demanded database blocks would be kept on faster storage to reduce access time. For example, imagine a sales compensation application that salespeople and their managers access constantly as they track their performance:

  • Depending on the sales cycle, it might be advantageous to pre-load the most recent quarter’s worth of data whenever the database instance restarted so that users don’t have to wait to report against the most recent data.
  • As the calendar year comes to a close and year-end bonuses are due, however, other database objects containing quarter-end summary data might be pre-loaded into memory as well.

Finally, IMDB – in concert with Oracle’s built-in Oracle database buffer cache management – also evicted data blocks singly or en masse when they were no longer needed for queries or DML. This memory management strategy kept only the “hottest” data blocks in memory nearest the compute infrastructure that needed them most.

GPUs Have No Buffer Cache. Enter Infinia.

Much like the database buffer cache mechanism I’ve described, DDN’s Infinia data engine is targeted specifically at optimizing inference operations.

Leveraging the open-source NVIDIA Inference Xfer Library (NIXL), Infinia runs on the GPUs themselves itself to provide a unified namespace for inference operations. In concert with EXAScaler, it captures and stores all LLMs and their metadata, the complete KV cache, and – most importantly – the related context about data within the KV cache.

DDN delved deeply into the advantages of Infinia for keeping data as close as possible to the GPU during inference operations, especially prefill. Infinia’s key differentiator is its ability to keep the CPU out of the data path through its zero-copy, direct memory access (DMA) to storage. In concert with GPUs – for example, with NVIDIA’s Dynamo framework – Infinia can halt prefill operations before a GPU in the cluster rewrites anything.

Overcoming Access Path Mismatches

As DBAs, we encouraged our application developers to write better SQL statements so the database query optimizer built the most efficient access paths to locate exactly the data needed as efficiently as possible – leveraging bind variables, knowing which columns were indexed for fastest selection criteria application, or even providing specific optimizer hints to clarify which methods to choose or ignore.

Advanced enterprise databases also provide analytic tools to identify which statements are the worst offenders during different workload patterns, making suggestions on improved performance by partitioning database tables, adding appropriate indexes, or gathering data frequency histograms to assist the query optimizer to make better decisions on access paths.

Unfortunately, AI agents are much more unpredictable than SQL statements when requesting data. They may request data via a completely different prompt, creating an access pattern mismatch and thus force a GPU to recompute when it’s totally unnecessary, and that leads to unscalable performance. Additionally, an AI agent may be requesting data in varied formats ranging from simple text to images, MP3s, MP4s, and other file-based data sources.

Tightening the AI Inference Gyre

The needle that Infinia needs to thread is to return only the data needed based on its required context as efficiently as possible to the requesting AI workload. Of course, efficiency varies depending on the perspective of the requester – is it time to first token (TTFT), or time per output token (TPOT), that matters most when measuring it?

The unfortunate reality is that AI has muddled expectations of performance and efficiency because we’ve progressed well beyond my most experienced end user showing up at my senior DBA’s cubicle to complain the system is running slow.

DDN did a deep dive into complete lifecycle of AI inference to explain how Infinia answers these challenges by keeping the data that agents need to answer queries as close as possible to the GPU without constantly overwriting the KV cache.

Since Infinia essentially retains context as well as data within its object store, it knows precisely where to retrieve the needed data without unnecessary recompute.

From my perspective as a curmudgeonly DBA, Infinia appears to be about as close as an in-memory enterprise database we’re going to get that will hopefully handle the insanely diverse demands of generative and agentic AI.