What Is Local AI Actually For?
📍 Where
TR310-2, 3F, TR Building (研揚大樓 AAEON Building)
National Taiwan University of Science and Technology (NTUST 台科大)
No. 43, Sec. 4, Keelung Rd., Da’an District, Taipei 106
台北市大安區基隆路四段43號
🎟 Free admission. No ticket or registration required — just walk in.
COSCUP is a free and open conference, and you don’t need to attend the whole event. If this topic interests you, feel free to join us just for this one-hour session.
Introduction
Everyone is talking about “AI” right now. Far fewer people are asking whether it has actually made their work—or anyone’s life—better.
I build this stuff for a living: an on-device AI appliance built around a Raspberry Pi 5 and AI HAT+, as well as mobile apps that run speech and language models locally using tools such as sherpa-onnx, whisper.cpp, llama.cpp, Gemma, and Style-Bert-VITS2.
Not in the cloud, but directly on the device—for people whose data should not leave it.
Most of what I’ve learned, however, is how local AI breaks.
Small models forget what you just told them.
Latency is noticeable.
A demo impresses someone once, and then never gets opened again.
So this is not a talk where I pretend to have all the answers. It is a space for people who are actually building with local AI and LLMs—or are simply curious about them—to share what they have tried, compare notes, and speak honestly about what is and is not working.
Some of the questions we might explore include:
Where has local AI genuinely made something better—and where have we simply bolted a chatbot onto something that was already fine?
What is still missing: better models, tooling, hardware, UX patterns, or something else?
What would you build if someone else created the component you keep having to rewrite?
What did you have to give up to run AI on-device—and which supposed limitations turned out not to matter?
This will be a hands-on, participant-driven session.
If you have built something using local AI, LLMs, ASR, TTS, or related hardware, you are welcome to give a short lightning talk or show a quick demo using the projector. Tell us what you have been working on, what worked, what failed, and what you learned.
It does not need to be polished or finished, and you do not need to prepare a formal presentation. Work in progress, experimental prototypes, failed attempts, and unfinished ideas are especially welcome. Even a few slides or a short live demonstration is enough.
After each sharing, we will open the floor for questions and discussion. If you do not have a project to present, that is completely fine—bring a question, an idea, an experience, or a strong opinion.
No conclusions are required. “It would be great if this existed” is a perfectly good outcome.
A Raspberry Pi 5 will be running fully offline on the table throughout the session, so whenever the conversation gets too abstract, we can poke at something real.
Everyone is welcome: people working with edge AI, on-device LLMs, ASR, TTS, low-resource languages, or hardware—as well as anyone with a strong opinion about whether any of this is actually useful yet.
Newcomers are especially welcome. You do not need to be an expert, and you do not need to bring a finished project.
Bring what you have built, what you have learned, or simply the question that brought you here.