Author: jim.czuprynski

I’ve heard from Cisco more than a few times at prior Tech Field Days. As AI continues to penetrate every aspect of IT organizations’ workflows, they offered some interesting insights on the future of monitoring network performance at AI Field Day #8.

When Gladys Isn’t Happy, Watch Out.

When I taught DBAs the advanced Oracle database performance tuning course, I often talked about a mythical user – Gladys – who knew how fast the company’s primary applications should be working better than anyone.

If Gladys wandered by and said “The database is running slow,” you knew you were probably in for a long day and had better pay attention to her complaint … or the next thing you know, your boss would be getting a call from the CIO.

And conversely, if Gladys gave you the thumbs up after a major deployment, you knew you’d done your job.

It appears the folks at Cisco heard of Gladys from their customers – so much so they built a slide around her nebulous complaint.

My point, of course, is it’s relatively simply to diagnose normal, non-AI workloads because network admins, DBAs, and DevOps folks are reasonably skilled at looking at pertinent metrics – I/O wait times, network ping round-trips, application audit logs, average database response time – that quickly point at likely culprit(s).

Nexus: Cisco’s Toolset At the Center of Agentic Operations

AI workloads, however, often defy normal troubleshooting because they span diverse endpoints, including GPUs deployed for intense training, or dedicated compute nodes intended for purely inference operations.

Cisco’s latest strategy for navigating this complex melange is called AgenticOps. At its heart is Cisco AI Canvas, a collection of network-specific models they’ve developed across decades of network monitoring and diagnostic experience, that can deploy an AI agent via MCP to handle AI-centric workload issues.

Cisco demonstrated their AgenticOps strategy concepts through their Nexus Dashboard tools. They showed an AI-assisted diagnosis of networking anomalies related to GPU performance in plain business language, without having to know any of the typical networking CLI commands I’d need to know to drill into what actually was wrong with the corresponding network components and their configurations.

Even better, the Nexus agent framework was able to compare the network configurations against “golden” configurations and recommend potential remediations. As I’m certainly no expert on BGP command line instructions, so I’d probably call in some favors from a network expert colleague before I proceeded to apply the fixes recommended.

I found Nexus offered some intriguing statistics during a demo of it within the Splunk Observability Cloud. Tied to a series of Cisco AI PODs, it’s even possible to view overall “tokenomics” for AI-related operations – for example, how much a single token cost much each POD actively costs in several dimensions, including how many tokens (both input and output) have been expended, cumulative token costs, average cost per request, and any other applicable metrics.

AI Canvas: Human-Agent Collaboration For Issue Resolution

Cisco’s final demonstration showed how their AI Canvas tool made it possible to bring an experienced network engineer into a dialogue with other engineers who were actively attempting to isolate a particular AI workload issue.

The tool was eventually able to help the engineers to draw accurate conclusions about what was actually affecting performance: two GPUs were apparently under-utilized because the network connections to those GPUs’ compute nodes were not working with nominal standards.

Cisco admitted the examples they demonstrated show promise, and they’re planning to expand their Nexus toolsets to their customer base in the future. From my perspective as a definitely network not-know-it-all, I have to admit that even this relatively early stage of their AgenticOps strategy is impressive. At the very least it would keep someone suddenly tasked with figuring out how to solve a complex network issue from acting on minimal intelligence and making an utterly foolish and incorrect decision.

As part of my first serious Oracle DBA assignment in 2001, the first production database I’d ever built was hosted on an HP8500 ProLiant server – at the time, a serious computing beast with 8 removable CPUs and an enormous 8GB of physical memory.

I spent plenty of frigid days in our server room building storage arrays, attaching servers to storage and networking, and keeping a close eye on the hardware to insure everything remained fully available and performant. And over the next five years, that ProLiant simply never quit, even when under stress of demanding OLTP and reporting workloads.

Agentic AI Is Inference-Intensive. Robust Compute Is Still Crucial

Of course, that HP8500 is now ancient technology – it would likely melt down if asked to handle 1% of today’s generative AI, RAG, and agentic AI workloads that have dramatically expanded the breadth of requirements for extremely reliable and performant compute resources.

The latest ProLiant series we saw as HPE presented at AI Field Day #8 recognize these new computing demands in several dimensions.

HPE’s Senior Principal Product Manager Bharath Venkant talked at length with us on the changing dimensions of AI compute, from on-premises thru public and hybrid cloud, plus the growing need for AI inference at the edge.

I found HPE’s orientation on AI refreshing: They realize there’s an incredible divergent breadth of compute requirements for customers as journey towards whatever AI destination they’re headed, and they’ve (wisely!) focused on meeting customers wherever they might be right now on that roadmap.

The HPE product continuum I reviewed at AIFD8 offers everything from a DevOps AI “starter pack” sporting two NVIDIA RTX Pro 6000 GPUs all the way up through a continuum of small, medium, and large computing setups.

And for the most extreme AI compute requirements, HPE offers an expansion rack option housing as many as 64 NVIDIA H200 (“Hopper”) GPUs.

HPE’s AI Engine strategy thus promises to simplify an IT shop’s their AI-specific compute needs as they traverse shifting training and inference requirements because ever-increasing demands can be met within an identical, reliable hardware framework.

Inference At Razor’s Edge Demands Reliable Compute

I’ve always thought of HPE focused on building servers for pristine data center racks, so I was pleasantly surprised to see their offerings for AI at “the edge.” Factory automation, warehousing, logistics, and biomedical production immediately popped into mind … but certainly not battlefield situations.

HPE built their ProLiant EL2000 2U server chassis specifically for extreme edge conditions; it’s rated for military-grade ruggedness and resilience and plugs into their EL220 and EL240 Gen12 servers.

Extreme conditions – vibration, temperature, dust, intermittent connectivity, variable access to power and cooling – meant HPE had to build compute components to accommodate some potentially challenging edge constraints.

And while AI workloads at the edge are more likely focused primarily on inference to interpret critical situations and leverage agentic AI to recommend immediate action with human oversight, I’d foresee that data collection for retransmission back to a “home base” for later model fine-tuning could also be a requirement.

Private Cloud AI: Blueprints Sure Help

While I’ve spent the majority of the past decade either presenting at conferences or helping clients build complex database solutions, I’ve found myself immersed in DIY home remodeling. Fortunately, my successes outnumber my failures, but I’ve noticed my most successful projects start with intense and detailed planning, especially a drawing of what I intend to create.

Unfortunately, it’s an open secret that many IT shops struggle with laying out plans – much less building – a comprehensive AI solution that fits their organization’s needs. Mark Seither, PCAI Senior Principal Solutions Architect, explained in depth HPE’s alternative to DIY AI: their Private Cloud AI solutions.

The smallest system available – aka their developer system – uses a DL-325 server as a control plane plus a DL-380A that hosts two 2 NVIDIA RTX Pro 6000 GPUs.

This gives system admins and their DevOps teams a chance to get familiar with HPE AI capabilities – what I like to think of as the targets the “poke it with a stick” orientation – before deciding which configuration is likely needed for current production demands.

Moving up the sizing scale, the medium and large configurations shift from RTX Pro 6000s to NVIDIA NVIDIA H200 (“Hopper”) GPUs, with the largest system accommodating 10 H200s. And as mentioned previously, expansion racks can be added that handle up to 64 H200s. (HPE also mentioned they’re planning a 128 H200 expansion rack in the near future.)

If I were tasked with building out a new AI platform strategy, I see several advantages of this approach:

• Since HPE deploys identical software across the entire platform family, it’s relatively simple to upgrade from, say, the developer system to a larger Private Cloud AI systems.
• Speaking of the value of planning ahead: HPE leveraged NVIDIA’s published blueprints to create the most efficient and lowest-cost options for immediate deployments.
• Finally, the partnership with NVIDIA offers advantages for AI implementations. NVIDIA’s open-source Nemotron 3 family of models, datasets, developer tools, and inference capabilities look pretty intriguing!

A Real-Life Example: Why Is My Discount Denied?

As an experienced developer, DBA, and presenter about technology, I’ve seen the value of the Don’t just tell me – show me maxim, especially when it comes to all things AI. I wasn’t disappointed with an all-too-brief demonstration of HPE’s quick-to-market AI capabilities.

Michael ran a live demo of their AI Essentials toolset to show how simple it would be to construct a chatbot that leveraged Langflow Agentic AI Workshop.

In under five minutes, he had deployed a reasonably sophisticated chatbot using the Nemotron 3 Nano LLMs from NVIDIA that accessed travel reimbursement policies retained in an internal document store. He then simulated requests from customers trying to understand why a particular travel refund request wasn’t handled to satisfaction .

How Do You Know Where You’re Going If You Don’t Know Where You Are?

So many IT shops these days are hearing the same drumbeat of their CIOs:

Look, We know {generative AI | RAG | agentic AI | predictive AI} is the solution – now go find problems to solve!

I found HPE’s orientation and roadmap for building a practical compute strategy refreshing because it was grounded in real-world use cases and offered a consistent vision for meeting a potential or current customer wherever they are in their AI journey, and their close partnership with NVIDIA adds stability and flexibility to whatever that journey may potentially lead.

I heard from the folks at Scality at AI Field Day #8 for the first time at a Tech Field Day event, and I was happily surprised to discover some new takes on storage management in this brave new era of everything Agentic AI.

Full disclosure here: I have 25+ years of Oracle DBA experience, and I’ve also built storage arrays from the ground up. I also spent two years as a database subject matter expert at Hitachi Data Systems; my experience there convinced me just how poorly most DBAs and DevOps folks understand storage technology, especially how to monitor it for optimum performance.

Scality’s approach to managing complex storage environments definitely captured my attention.

Storage Performance Tuning Isn’t Easy. Why Let Customers Fumble Through It?

Scality’s CTO Giorgio Regni introduced their approach to modern storage strategies with a clever but refreshing history lesson on how most storage providers have tried to satisfy their customers’ demands for visibility into their storage platforms over the past decade or so:

• First, customers demanded more control over their storage environments, so providers created complex control structures with a multiplicity of “buttons” anyone could press in hopes performance would improve.
• When that didn’t work, providers created overladen dashboards that hopefully highlighted what was actually non-performant – as long as you knew exactly where to look at the right time.
• As dashboards proved a distraction to immediate action, providers built complex alert mechanisms that triggered red flags – often sent right to your mobile phone, because who doesn’t want an alarm at zero dark thirty to fix a failed device?
• Finally, providers decided the real solution for tuning storage performance was to give customers the power to configure anything they wanted through complex config files.

Giorgio’s point? These approaches have given humans complete control over their storage architecture, but unfortunately that meant humans have become the control plane. And that’s trapped system reliability engineers, storage administrators, and maybe even the occasional unfortunate dinosaur DBA who’s their 3rd-tier backup into making decisions to improve performance that exceeded the needed expertise.

Obviously, this approach is unsustainable, especially in light of the complex modern AI systems’ demand to provide peak performance of the underlying storage, whether those workloads are intense inference, long-running model training, or a mixture of both.

An Agentic Approach: Scality ADI

Scality chose AIFD8 as their venue to announce the release their latest product – Scality ADI, short for Autonomous Data Infrastructure. Essentially their product provides a single view of the totality of an enterprise’s storage resources as if it were a single AWS S3 storage endpoint.

While this approach to solving storage flexibility isn’t necessarily dramatically different from some of their competitors’ offerings, what intrigued me was their implementation of their agentic AI assistant, Guardian.

A Scality ADI user can leverage the Guardian agent UI directly to perform day-to-day standard storage management operations, handle data security operations, and even perform storage tasks an under-experienced user has limited or no knowledge of how to issue the appropriate commands. Scality also enables more sophisticated IT shops who have already begun to embrace agentic AI to construct their own storage management UI via MCP calls.

Sure, I Could Master Storage Needs If I Just RTFM’d. But Which FM Do I Need To R?

But the brief demo of Guardian that Scality performed at AIFD8 helped me understand how I could leverage Scality ADI to perform storage management without knowing what specific commands to issue.

Whichever method chosen to implement it, Scality ADI encapsulates 15+ years of storage management experience into a single AI-driven toolset that lets the least experienced user leverage an intelligent set of tools that will invoke the proper commands to perform complex operations without knowing the microscopic knowledge often required. And regardless of the implementation chosen, Scality ADI keeps a human in the processing loop at all times to make sure nothing incredibly foolish accidentally gets executed.

Any company who presents their first whiteboard image for their product at an event like AI Field Day #8 is sure to capture my immediate attention.

This was the first time I’d heard from Selector AI at a Tech Field Day event. I found them to be gutsy and independent while offering unique solutions to figuring out network issues in modern AI system architectures.

A Man’s Got To Know His Limits

Some background: I’m a reasonably experienced Oracle DBA, so I know how important a reliable network is for applications to connect to my databases.

I also know my limits.

I know I’d never be able to fill in as a backup network administrator during an irritating recurrence of a periodic latency, and definitely not during an unexpected crisis. I see the value of Selector AI’s platform as insurance against me doing something stupid after I guessed at a solution based five minutes’ research on Stack Overflow.

It’s All Fun & Games … Until Somebody Loses a Packet

Finding the true root cause of a network performance latency often becomes a multi-vendor finger-pointing exercise, especially when up to that point in time everything appeared to be working nominally.

When you factor in the obvious – no two IT shops’ network infrastructure is truly identical, even if they used identical hardware – it can be nearly impossible to assess whether sudden latency is expected because of normal business work schedules, versus the failure of a critical hardware component or because a newbie engineer deployed an untested DNS configuration.

That’s why Selector AI built their monitoring platform with the ability to capture the specific context of their client’s network performance during what would be considered normal and acceptable.

Selector AI’s platform can then deploy bespoke monitoring via proprietary AI models they’ve developed to filter out the noise from millions of log posts produced from many thousands of network devices to isolate root causes effectively.

When A Single-Shot Solution … Isn’t.

Joby Rudolph, Selector AI’s distinguished engineer, demonstrated in detail how the most recent version of their platform came into being.

Their first version used a single-shot AI approach to solving network issues; it enabled an unskilled human to ask questions using a proprietary NLP chatbot approach to ask simple questions about the network’s state. But as more robust AI infrastructure has matured in the past 18 months – especially the capabilities of Model Context Protocol (MCP) tools – Selector built the latest version of their platform around that orientation.

They realized the models already developed for detecting network issues were still valuable, but they migrated their solution from single-shot inferencing to leverage an agentic approach instead.

This latest version deploys three different types of agents to diagnose and solve a non-performant network:

• An orchestrator agent coordinates the activities of all other agents in the stack.
• A series of domain-specific agents tackle tasks across the network – for example, querying the health of an individual switch – and then report results cohesively to the orchestrator.
• Domain-specific agents then leverage one or more MCP agents to obtain the results from the network desired component.

During an (all-too-brief!) demo, Selector AI showed how a detected problem (shown as a non-green hexagon in main monitoring UI) could be drilled into and then queried in natural language to provide advice on how to fix a particular issue.

Need Tools? Great. Got Tools? Perfect!

The three-level agentic model lets Selector AI offer bespoke solutions to each client in their portfolio, meeting them client where they are right now in their network problem resolution methodology:

• As some clients already have a well-defined toolset for solving network issues, the Selector AI platform acts as an orchestrator to apply those tools directly.
• Alternatively, their platform also lets a client who already has their own orchestration and solution toolsets to leverage the Selector AI models to figure out what’s malfunctioning and – always with a human in the loop where necessary – solve the problem with more precise intelligence than if the client had built their own detection and severity ranking infrastructure.

Selector AI offers a brief demo of its platform’s offerings that was roughly analogous to what they showed us, so grab a look so you can see it in action.

The AI continuum lately feels as if it’s being warped by the sudden appearance of a hidden and super-massive black hole. Spiraling development costs have given rise to feverish discussion of tokenomics as IT organizations struggle to limit their DevOps teams from depleting a year’s worth of tokens in just a few weeks. But runaway token spends are just the most visible part of today’s AI challenges.

Again With the Data.

Our final presenter at AI Field Day #8 – Hammerspace – targeted a key dimension that IT shops must focus on when building robust AI solutions that hopefully will yield meaningful results for their applications’ end users: AI still requires humongous volumes of data to produce accurate intelligence.

Running Out Of Everything Everywhere, All At Once

Hammerspace succinctly summarized the multiple uncomfortable realities about AI resource availability being discussed in just about every C-level boardroom discussion these days:

• Datacenter power resources are crimped because utilities are unable to increase capacity quickly enough.
• Even if you could even buy more processors, CPUs continue to be expensive.
• Since storage is mainly SSD- and NVME-focused, the availability of reasonably priced storage is limited, too.
• Finally, if a shop cannot secure additional power, compute, or storage within their own confines, cloud capacity is no longer a guaranteed off-ramp because nesoscalers – and even the largest hyperscalers – are nearing the maximum limits of their commodity hardware, just to support the existing workloads of their other customers.

Unifying, Rather Than Just Consolidating, Data

Hammerspace’s AI Data Platform solution thus implements a data unification strategy, rather than merely applying typical data consolidation approaches.

When a request is made for a specific set of data – for example, several thousand documents needed for additional AI training, or a vector-driven similarity search during an intense inference operation – the Hammerspace AI platform gathers those resources into the Tier 0 layer so operations can complete as quickly as possible. And when operations cease to require speedy access to resources, they can be moved intelligently to other (s)lower storage tiers.

Their solution essentially builds a global namespace that appears as a single storage viewpoint. Their platform stands in front of an organization’s present set of storage clusters and can access data from anywhere, regardless if it’s retained in local SSDs or NVMes – what we typically call Tier 0 – or an on-premises private cloud, or even a public cloud.

The advantage of this strategy? It treats data, wherever it might exist within multiple physical storage layers, as if they were kept within a single massive cluster.

Storage Vendor Partnerships = Even More Interesting Use Cases

Hammerspace’s customer base illustrates the attractiveness of their platform – they’ve implemented within customer HPE solutions as well as neoclouds. And Hammerspace also described several quite divergent use cases, including an AI application development environment supporting data retrieval demands spanning multiple petabytes and a user base supporting thousands of data scientist end-users.

Hammerspace also co-presented on how their global storage model helps Hitachi Vantara expand the capabilities of its VSP One platform for several of its AI-focused customers.

In concert with Vantara, Hammerspace’s solution was deployed across intriguing use cases, including a complex AI-based fraud detection and risk assessment application, as well as and an AI “incubation hub” where the Hammerspace solution sat above VSP-One block storage and communicated with an NVIDIA-powered two-node GPU cluster.

(Full disclosure: I’d actually helped launch the first iteration on the VSP in 2010 with extensive experimental research on how best to use Hitachi storage solutions to handle extreme Oracle database workloads as they consumed SSD storage bandwidth. I remember one workload was so intense that we actually melted a few SSDs in their cases.)

Storage Has Always Just “Been There”

If I were back in my role as a senior guy “in the trenches” just trying to keep a team of data engineers and data scientists productive and frustration-free, I can see exactly how valuable Hammerspace’s unified storage approach would benefit us. At every IT shop I’ve worked at, the expectation is that the storage layer is transparently ubiquitous, works all the time, and never ever runs out of space.

Of course, that’s a fantasy, and with the onset of increasing AI workload demands like training and inference that are often in direct conflict with each other, anything that helps homogenize and simplify managing storage is worth a closer look.

Our team didn’t get to see how Hammerspace’s AI Data Platform worked in concert with Hitachi VSP One because our final session ran out of time. That’s not a bad sign – it just means our team of delegates found lots of great questions to ask; check out the introductory video to see we all found interesting.

You know what I dearly love about attending Tech Field Day events? They aren’t just about humdrum, boots-on-the-ground problems we deal with every day. I found myself re-energized about what IT technology can accomplish for humanity at large after hearing more about the history of what we call The Cloud; tireless efforts to preserve trillions of web pages since the Internet’s inception; and just how tiny and precious our pale blue dot is when compared to the vastness of the universe.

The Cloud: A Brief History of Irrational Exuberance

Tom Lyon, an historian of computer science who has been programming since the early 1960s and co-founded DriveScale, stepped up to give us a totally different perspective versus how most of us delegates have probably thought about the current state of all things Cloud – public, private, hyper-scaled, neo-cloud, you name it – but posited that what we call The Cloud these days actually has its genesis nearly a century ago.

The interesting part? We’re really just ending yet another cycle of what Tom posits he and other keen observers call irrational exuberance as the famed AI hype cycle plays out.

Tom noted hyperscalers like Amazon, Google, Oracle, and Microsoft have built a house of cards around unbelievably rosy projections of AI growth possibilities, furthered by complex accounting tricks like special purpose vehicles to hide debt on their balance sheets. I commented that I’m just waiting for some 70-something investor to pop up at a shareholder’s meeting to shout at Larry Ellison, Satya Nadella, or Sundar Pichai Excuse me, but are you on crack?

The Real Ministry of Truth. Apologies to Winston Smith.

The continuing degradation of objective truth in the modern world has severe implications for policy-making, corporate planning, and history at large. It’s impossible to ignore the erasure of large swaths of scientific data about everything from climate change to basic public health that continues in real time.

Without the Wayback Machine, many USA government websites’ content that’s been effectively erased by the current administration in Washington. That’s why it’s so crucial to preserve the Internet’s billions of pages of content created over the last 40 years since the World Wide Web was born.

We heard from Joy Chesbrough, the Internet Archive’s Chief Philanthropy Officer (now that’s a cool job title!), on how their organization is able to preserve the past digitally on a relatively tiny budget of $30M per year. It was inspiring to hear how this non-profit is able to do the sometimes-thankless work of digital preservation and what those efforts conserve for future generations.

We’ve all seen how governments, politicians, and oligarchs have attempted recently to discard uncomfortable truths – everything from the denigration of the 1619 Project’s attempts to document how slavery played a key part of the founding of the American Experiment to recommended standards for nutrition, vaccine scheduling, and disease prevention. The Wayback Machine is a crucial tool for preserving that treasure trove of human knowledge against the creep of religious conservatism, willful ignorance, and authoritarian governments.

There’s No Intelligent Life Here. Do Look Up!

It’s easy to get discouraged about the current state of humanity, but I found my faith restored during our visit to the Search for Extraterrestrial Intelligence (SETI) Institute in nearby Mountain View.

We took a tour of the SETI offices after we chatted with Dr. Christina Ricci about the humongous scope of the universe and the mediums they’re searching for evidence of industrialized and advanced civilizations. She explained the famous Drake Equation which explains just how likely we tiny humans are likely to find evidence of extraterrestrial intelligence, if we just look hard enough. It might even happen in our lifetimes … and it would utterly transform the way humanity thinks about itself once we know we are truly not alone in the universe.

Dr. Vishal Gajjar explained how his efforts at SETI deploys artificial intelligence – really, complex machine learning algorithms trained across exabytes of radio and visual astronomy – are being actively applied to determine if there really are advanced civilizations in our galactic backyard. One of the most fascinating ideas involves looking for evidence of construction of mind-bogglingly huge structures like Dyson spheres surrounding exoplanets and stars across years of collected data in both EM and visible light spectrums.

Of course, I had to buy a couple of t-shirts to support their efforts. 🤯