[2/3] AI Leadership Summit 2025 - AI Leadership (AI column)
Continue. story about this conference, I will share the next batch of reports
Booking.com**: Agents With Real ROI (**02:28:27) Bruno Pasos of booking.com and Byang of Sourcegraph We talked about the introduction of AI agents in booking.com. The original KPI was the number of hours saved using Gen AI, then the guys experimented with the approaches and refined the KPI. So we got about the following picture. - Lead time for change (short term): time to review, time to debug (error log summary), time to merge - Quality (mid term): vulnerability reduction, test coverage, reduce change fail rate - Codebase insights (mid term): track dead code, not performant (redundant queries), replatforming - Tech modernization (long term): monolith reduction, feature flag cleaned (In booking, because of the large number of experiments such flags are very many.) The guys made the GraphQL Query Generator, as they have a fairly large and weighty surface GraphQL API. They also worked on automated code migration and got good results. They also worked to improve the Code Review. By the end of the year, all of this has helped developers build on the 30% merge requests, and a key success factor was training developers to work with these tools.
Writer: Evaluating LLMs for Financial Scenarios (02:49:31) Wasim Alik of Writer presented the results of the LLM assessment for financial scenarios. They created a FAIL dataset to evaluate models in real-world scenarios with query errors and context. The study found that even the best “thinking” models often produce responses with high levels of hallucinations when the context is incorrect or absent. Alik concluded that specialized domain models are still needed, despite the high accuracy of common models.
OpenAI: OpenAI for VPs (03:01:34) OpenAI spoke about the strategy of implementing their technologies in enterprises of three phases Creating an AI-literate workforce (via ChatGPT) Automation of internal operations (via API) Integrating AI into end products They stressed the importance of aligning AI strategy with overall business strategy.
OpenAI: Building Agents the Right Way (03:09:47) OpenAI spoke about the right approaches to creating AI agents, where there are three key components:
- Reliable context engine High-quality model of reasoning Secure code execution environment Examples of successful implementations of agents in enterprises where routines were automated and efficiency increased were presented.
Frontier Feud (04:25:41) Game-shaped panel 100 toward 1Where to guess the answers to the questions. It all started with the most famous AI scientists, and then moved on to "top considerations when choosing a model" and so on. In general, the activity was purely fanatic for the participants, but about useless for the audience.
Reuters: Missing Pieces of Workflow Automation (04:51:29) Reuters representatives shared their experience in implementing AI to automate workflows in the media industry. They identified the key “missing elements” of successful automation: integrating with existing systems, training employees, managing expectations, and creating clear success metrics. Special attention was paid to how AI helps journalists process large amounts of information and create quality content faster.
Arize: Evaluating Agents (05:06:07) The report was about the methodology for assessing the effectiveness of AI-agents, which is actually the product of Arize. The speaker began with a talk about closing the C round of funding, and then presented various metrics and approaches to testing, including assessing accuracy, speed, error resistance and ability to follow instructions. The rapporteurs stressed the importance of establishing integrated assessment systems that took into account both the technical aspects of the work of agents and their practical utility to end-users.
Concluding the first day of the summit in last post.
#AI #Software #Product #Management #Leadership