A UCSC Google Account is required to access this tool.
Brace Tools
Brace is an LLM-powered course assistant created by Professor Adam Smith to help with teaching feedback-intensive courses with large student populations. Unlike other LLM-based assistants, chats with Brace pull in instructor-authored content as needed in order to carry out activities and behaviors specific the the course. Brace integrates with the Canvas LMS in order to allow students to directly submit conversation transcripts.
What? It lets you have conversations like the one in this transcript from an early prototype of the system. It's ✨agentic✨.
How are students reacting? Here's a summary of student
reactions to a first assignment that
Where can I audit the source code for Brace? The source code for this system can be found at https://github.com/rndmcnlly/brace. Because Brace can pull assignment descriptions and other text from Canvas, more and more of Brace's behavoir is controlled by natural language instructions that are private to the community of the course and not visible on GitHub.
Chat with Brace
System Status
Design Notes
FAQ
- Who pays for all this? It's just Adam Smith building and operating this thing. We spend about $12/month for the front-end server and $1/student/month for the back-end LLM service. Roughly, thats 20k input tokens (mostly cached) and 1k output tokens per student per day. We expect costs to drop to $0.25/student/month very soon.
- Why is this on the
brace.toolsdomain and not something more official likebrace.ucsc.edu? It takes many meetings and approvals set that up. I'm working through it. - Where is the data stored? Conversations are stored on Adam's server, and copies of some of them are submitted to the Canvas LMS via explicit student actions. Conversations may be temporarily stored by back-end LLM providers to speed up future text completion calls.
- Who can access student conversations? The instructor is the only one who can access student chats on the server (and students cannot look at peer work, and TAs can only see submitted chats).
- How is this different from ChatGPT? A few important ways:
- The system disguishes the roles of students and teachers and their relation to a specific course that organizes assignments.
- In a typical ChatGPT conversation, 80%-90% of the context window is filled with recycled AI-generated text. With Brace, about half is populated with instructor-written text specific to the course, and student code pulled from public repositories on GitHub further grounds the conversation in the student's work.
- When the LLM makes an embarassing mistake (e.g. regarding course content or pedagogical technique) the course staff can immediately fix this at the course level by editing the assistant's system prompt, wiki pages, or text on Canvas.
- Access to the top models or most comfortable rate limits is not gated by a student's wealth. All students get the best experience for free.
- Conversations stay within the course, and they are not sold or mined for others' commercial/institutional benefit.
- What data is used to train Brace? Brace has not been trained specifically for use in this course, and it only knows about the course via prompts (i.e. text injected into the LLM's context window). Brace works as a wrapper around pre-existing LLMs (which are each trained on their own peculiar and likely problematic sources of data).
- Which LLM is Brace based on? Brace was developed using the
gpt-4omodel as the default back-end. However, it is easy for us to swap in alterantives. We are looking for alternatives that are strong (less likely to misbehave), not too costly, and less energy intensive. If monetary cost were no concern, we would still prefer a moderate-strength model to show that the largest models are not necessary for this application. See Artificial Analysis for a report on the rapidly changing landscape of commercially-available LLM models and services. - Where can I listen to the Brace soundtrack? Enjoy these musical design-fiction tracks we created to keep us hyped during the system's development: 🎵 Brace playlist on Suno.
What's new with Brace Tools?
- offering students a zero-cost LLM assistant
- offering students a course-specific LLM assistant
- agentic consultation of pages from a course-specific content+pedagogy wiki
- agentic review of student code pulled from public GitHub repositories
- dialog-based first-time setup of communication preferences (e.g. picking non-English languages)
- dialog-based quiz activities with integrated feedback
- dialog-based submission for programming projects (git precheck)
- submitting work directly to Canvas (without manual artifact download/upload)
- allowing students to log into off-Canvas tool using SSO (no passwords, via Google OAuth2)
Related university-provided LLM chatbots
- CS50 Duck at (notably the only course-specific assistant on the list so far)
- WesBot at UC Santa Cruz
- U-M GPT at University of Michigan (see also Educause coverage)
- TritonGPT at UC San Diego
- ZotGPT at UC Irvine
- GPT at Washington University
- Paul AI and GodaAI at Vilnius University