From Data to Intelligence: Real-World AI Systems on the Lakehouse
Summary
Join us for a series of technical talks focused on practical approaches to building, integrating, and operationalizing AI and machine learning systems using Databricks and modern data platforms. This event brings together engineers and data practitioners to explore real-world implementations—from creating AI agents to developing and deploying large-scale recommender systems.
Schedule
18:00 Doors open
18:30 - 19:00 Building a Recommender on the Lakehouse: An End-to-End MLOps Architecture (Boyang Yue)
19:00 - 19:30 Taking Machine Learning Models from Research to Deployment (Waylon Chen)
19:30 - 20:00 Building and Bridging AI Agents with Databricks (Shuji Oya)
20:00 - 21:00 Networking
21:00 Event ends
Talks
Talk 1: Building a Recommender on the Lakehouse: An End-to-End MLOps Architecture
Speaker: Boyang Yue
Abstract: Delivering personalized recommendations to over 70 million users requires more than just powerful algorithms; it demands a robust, scalable, and automated data intelligence platform. As the largest mobile payment provider in Japan, PayPay is committed to continuously improving the user experience for every customer. This presentation will outline how we build the end-to-end architecture to power our core recommendation system: PayPay Brain. We will walk through the entire machine learning lifecycle, from architecting data ingestion and preparation, to developing and training models, and finally to deployment and serving. Beyond the technical implementation, the talk will distill our lessons learned in building a sophisticated ML system within a regulated fintech environment. We will discuss key architectural decisions for breaking down data silos between teams, implementing robust governance and security without sacrificing agility, and maintaining operational efficiency.
Bio: Boyang Yue is a software engineer with a master's degree in computer science. He joined PayPay in 2023. His interests include data engineering, recommender systems, high-performance computing, and functional programming. When he's not at his keyboard, you can find him exploring new hiking trails or learning new skills.
Talk 2: Taking Machine Learning models from research to deployment
Speaker: Waylon Chen
Abstract: We will discuss how we streamline the process of taking ML models and their dataflows from research to production with Databricks at PayPay. From a DS/MLE perspective, we will walk through how we develop our ML models at scale with PySpark and Jobs/Workflows within Databricks' ecosystem, focusing on the pain points abstracted away by Databricks. We will also share the simplified process to seamlessly automate deploying an ML model into the production environment with Databricks Asset Bundle.
Bio: Waylon is a Senior Data Scientist at PayPay Corporation, leading credit modeling development works. Since joining PayPay in May 2023, he has built various models for PayPay’s lending business. He has been pioneering model development and leading the implementation of end-to-end ML workflows with Databricks at PayPay.
Talk 3: Building and Bridging AI Agents with Databricks
Speaker: Shuji Oya
Abstract: This presentation will cover how to use AI agents with Databricks, focusing on two main approaches: Building and Bridging. First, we will explain how to build AI agents directly on the Databricks platform. We'll show how you can use your own data to create these agents, with options for no-code, low-code, or fully code-based development.
Next, we will discuss bridging, which involves connecting external AI agents to data managed within Databricks. This part of the talk will highlight the open nature of the platform. By covering both methods, we will demonstrate how Databricks is designed to help you avoid vendor lock-in, giving you the flexibility to choose the best tools for your data.
Bio: Shuji Oya is a Solutions Architect at Databricks, focused on the digital native industry. With his prior experience at several companies in this space, such as Yahoo and Google, Shuji is passionate about understanding core client challenges and delivering impactful business solutions.
Organizers
Shahboz Ibragimov: A Sr. Business Development Representative at Databricks Japan, where he drives data and AI transformation across Japan’s fast-growing digital-native companies - from gaming and fintech to SaaS and startups.
Outside of work, he’s a passionate photographer and videographer who loves capturing beautiful scenery across and beyond Japan.
Yizhou Fan: An Engineering Manager at PayPay, where he leads AI, Machine Learning, and Data Science initiatives within the Data Insights Department. Before entering the fintech industry, he spent several years in the self-driving industry at Woven by Toyota. Yizhou holds a Ph.D. in Computer Vision. Yizhou is also a licensed association football referee.
Ilya Kulyatin: Fintech and AI entrepreneur with work and academic experience in the US, Netherlands, Singapore, UK, and Japan, with an MSc in Machine Learning from UCL.
Supporters
TBC
Our Community
Tokyo AI (TAI) is the biggest AI community in Japan, with 2,500+ members mainly based in Tokyo (engineers, researchers, investors, product managers, and corporate innovation managers).