San Francisco-based startup Weights & Biases is launching two new capabilities aimed at helping organizations build and monitor machine learning (ML) models. These new features are designed to make it easier for enterprises to get AI models running effectively for production workloads. Weights & Biases is a platform that provides tools for AI/ML development visualization and workflow operations for supporting and developing large language models.
W&B Weave: A Tool for Customizing AI Applications
Weights & Biases has launched a new tool called W&B Weave, which is aimed at understanding models and data in the context of a visual, iterative user interface (UI) experience. Weave is a toolkit containing composable UI primitives that a developer can put together to make an AI application. Weave is also about user experience; it can help data scientists develop interactive data visualizations.
Weave is a big piece of Weights & Biases’ roadmap, and it has been a core part of how the company has been building out its platform. The tool has been used internally to develop the Prompts tools that were announced in April. Weave is the foundation that enables the new production monitoring tools as well.
Weave is being made freely available as an open-source LLMOps tool, so anyone can use it to help build AI tools. It is also integrated into the Weights & Biases platform so that enterprise customers can build visualizations as a part of their overall AI development workflow.
W&B Production Monitoring: Customizable Metrics Tracking
Monitoring an ML model is crucial, and that’s where Weights & Biases’ production monitoring service comes in. The production monitoring service is customizable to help organizations track the metrics that matter to them. Common metrics for any production system are typically about availability, latency, and performance. With large language models (LLMs), there are also a host of new metrics that organizations need to track.
Given that many organizations will use a third-party LLM that will charge based on usage, it’s important to track how many API calls are being made to manage costs. With non-LLM AI deployments, model drift is a common monitoring concern, where organizations track to identify unexpected deviations over time from a baseline. With an LLM using generative AI, model drift cannot be easily tracked.
For a generative AI model used to help write better articles, for example, there would not be one single measurement or number that an organization could use to identify drift or quality. That’s where the customizable nature of production monitoring comes in. In the article-writing example, an organization could choose to monitor how many AI-generated suggestions a user actually integrates and how much time it takes to get the best result.
Monitoring can potentially be used to help with AI hallucination. Retrieval-augmented generation (RAG) is an increasingly common approach to limiting hallucination, which provides sources for a specific piece of generated content. An organization could use production monitoring to come up with a visualization in the monitoring dashboard to help get more insights.
Weights & Biases’ latest offerings, W&B Weave and W&B Production Monitoring, are key additions to its platform, aimed at helping organizations build and monitor machine learning models. Weave is a new tool designed to make it easier to understand models and data in the context of a visual, iterative UI experience. Production monitoring is a customizable service that enables organizations to track the metrics that matter to them.
With its platform, Weights & Biases is providing a toolkit for AI development visualization and workflow operations for supporting and developing large language models. The ability to build and monitor ML models is crucial in the AI lifecycle. Weights & Biases is making it easier for organizations to do both, providing a valuable service to businesses looking to leverage the power of AI.
Leave a Reply