I was asked to build a Slack app in 2 days using AI – Here’s how it went

Share:

Experimenting with AI-Assisted Development: Lessons Learned

I was tasked to build a Slack bot with a specific goal in mind: testing how helpful AI could be in the development process. The bot itself was relatively simple – users can type a slash command to sign in and out of work for the day, notifying their colleagues when they are around.

Using Visual Studio Code as my IDE, I worked with Claude 3.5 Sonnet and GPT-4o, integrated as part of GitHub Copilot and standalone tools, to explore their capabilities in real-world development. What I discovered was a mixed bag of modest successes and notable frustrations – a journey that revealed both the current limitations and potential benefits of AI in development.

The Good: Breaking Down Problems

The AI proved useful when handling specific, well-defined tasks. When I broke down the project into smaller components, it could sometimes provide helpful suggestions. For example, when working with message parsing or response formatting, the AI offered code snippets that served as decent starting points, though they usually needed some refinement.

The Bad: Hallucinations

Coding with AI assistance came with significant frustrations. The most problematic issue was AI “hallucination” – where the AI confidently suggests solutions using non-existent tools or outdated methods.

In one memorable instance, the AI insisted I use a library which does not exist and provided detailed implementation code for it. I spent a few hours trying to get the library to work before realizing it was completely made up. Similar issues occurred with supposed Slack API endpoints were not real.

I often found myself spending more time verifying and correcting the AI’s suggestions than I would have spent reading through the documentation and writing the code myself.

Reality Check: AI as an Occasional Helper

Through this experiment, I found that AI is best used sparingly and selectively. Here’s what I discovered:

1. Limited Use in Planning While the AI could suggest features and approaches, I often found traditional planning methods more reliable. The AI’s suggestions sometimes overcomplicated simple problems or proposed unnecessary features.
2. Focused Code Generation Rather than requesting complete implementations, I got better results asking for specific, isolated functions. For example, “How would I write a function to validate Slack event signatures?” worked better than asking for larger components. However, even then, I often needed to modify the code.
3. Code Review and Optimization The AI could sometimes spot obvious issues in code snippets, though its suggestions for improvements weren’t always practical or necessary.
4. Documentation Help One area where the AI proved somewhat useful was in helping draft initial documentation and code comments.

Best Practices I Developed

Through trial and error, I developed these guidelines for AI-assisted development:

• Always verify library suggestions against npm/pip or the relevant package manager
• Test AI-generated code in small chunks rather than implementing large blocks at once
• Keep an eye out for deprecated methods or outdated patterns
• Use AI-generated code as rough reference material, not as a reliable starting point
• Double-check all API-related code against current documentation
• Know when to step away from the AI and just code it yourself

The Final Verdict

Building a Slack bot with AI assistance revealed both possibilities and limitations of current AI coding tools. The experiment showed that AI can contribute to development in specific, bounded contexts – particularly with isolated tasks like generating initial documentation or suggesting code patterns. However, it also demonstrated that AI is far from being a comprehensive development solution.

The most practical approach emerged from learning when to use AI and when to rely on traditional development methods. For straightforward tasks or when working with familiar APIs, direct coding often proved more efficient than going through the AI feedback loop. In fact, the time spent getting AI to work effectively – verifying suggestions, correcting hallucinated solutions, and testing non-existent libraries – significantly exceeded the time I would have spent building the bot traditionally. What could have been straightforward development stretched into a longer process due to the overhead of AI assistance.

However, this investment in understanding AI’s capabilities wasn’t wasted time. The knowledge gained from this trial – learning when AI is genuinely helpful versus when it creates unnecessary complexity, understanding its limitations with API specifics, and developing a workflow for effectively verifying its suggestions – will likely reduce development time in future projects.

The key takeaway from this experiment isn’t about whether AI is good or bad for development – it’s about understanding its current capabilities and limitations. Success with AI assistance comes from knowing how to integrate it into existing development workflows, rather than trying to make it the centrepiece of the development process. While AI tools continue to evolve, they currently serve best as one of many development tools, each with their appropriate time and place.

 

Note: This article was written by Darcy, one of our super versatile developers at Code and Visual. He usually specialises in game development but bravely took on the challenge given to him.

Share: