Scaling documentation without scaling your team: Dagster’s AI-powered strategy

Published

The original version of this article can be found on the All Things Open blog.

How smart information architecture and AI assistants help a small team answer 16,000 community questions a month.

In Enabling Community Education with a Little Help from AI, Dagster Labs developer advocate Colton Padden shared how a small team supports a fast-growing open source community around Dagster, a declarative data orchestration framework.

After receiving feedback from the community about missing and unclear documentation, the team ran a full content audit and rebuilt Dagster’s information architecture around the software development lifecycle. They rewrote docs, migrated to Docusaurus, and layered content for different personas: reference docs, copy-pasteable examples, real-world open source pipelines, blogs, an e-book, and courses that have already seen tens of thousands of completions.

Colton demonstrated how AI eased the team’s processes. With clear contributing guidelines in the repo and docs that live side by side with code, tools like Claude Code and other LLMs can generate first drafts of tutorials from implementations, review outlines, and translate material across mediums. Additionally, a custom “Ask AI” assistant, backed by Dagster docs, GitHub issues, and discussions, now answers more than 16,000 community questions a month and provides a fast feedback loop on missing content.

Key takeaways

  • Treat documentation and information architecture as core infrastructure – Clear, lifecycle-oriented docs, examples, and courses serve both humans and LLMs, making every other education effort more effective.
  • Build tight feedback loops with your community – Use GitHub, Slack, analytics, and AI assistants to listen for where people are getting stuck, then continually refine content based on real questions and blockers.
  • Use AI and collaboration to scale a small team – Well-designed CONTRIBUTING guidelines, monorepos, and AI tooling can turn engineers and community members into effective documentation contributors, with humans still providing the final review and voice.

Ultimately, Colton argues that in an AI-driven world, high-quality educational material and empathetic listening to users are more important than ever. AI can scale a small team and empower contributors, but it’s only as good as the documentation and community practices behind it.