Developing nudger

As part of our internal AI group, we wanted to solve a (our) real-life problem and not just build yet another agent hello world. We have an internal board with internal tasks that come up in our regular retro. But since they are not our daily/project work, people tend to forget about them.

Nudger agent

That's how we came up with the idea of an AI agent that nudges people to finish those internal tasks. It reviews tasks, checks if they are due, and checks the comments to see who is blocking them (e.g. someone needs to review it). Then it starts reminding people what is needed from them and asking for expected timelines. Sometimes it has to repeat the requests, but tries not to be too pushy ;-)

Development

Nudger has 2 main responsibilities:

• Data Gatherer looks up old/stale stories and identifies the people who can help or are blocking a story.
• Talker nudges/chats with the people involved. We ran a separate Talker instance for each conversation.

We use LangGraph so that we can orchestrate subagents more easily. The whole graph consists of multiple LangGraph nodes, where some of them used an LLM and some implemented (old-school) deterministic logic without any LLM magic.

While Data Gatherer was instructed via system prompts on how to find old stories and responsible people, reality showed that we needed a smarter model to do the job. This was not a problem, as it ran only once a day.

Talker was executed more often, as it needed to check whether there were new replies, or simply to check the progress of a story based on the recent conversation with users. Here the entrypoint node was implemented without an LLM. One reason was token usage - we didn't want to burn tokens just to find out that no action should be taken. The second reason was that here we wanted stricter logic about when it makes sense to talk to a user (don't talk outside business hours, remind yourself after 2 days or when the user promised the job would be done, etc.), and we had seen that Data Gatherer occasionally hallucinated and did not always follow instructions.

MCP servers

This time we also needed MCP servers to give the LLM agents the ability to read our issue tracking system and to chat with users. The nice thing about MCP is that there are plenty of servers available.

For issue tracking, we just wanted to filter stories and read their details (comments etc.). We used an available off the shelf MCP server. The output was not nice (lots of things we didn't need), but it somehow worked (read the anecdote later in the testing section). We provided read-only access - so the agent could not do anything wrong.

For Google Chat it was more complicated. It was not an ordinary chatbot, nor someone talking on our behalf (it has its own identity). And it was not "read-only" - so we had to make sure the agent did not go rogue and do any unnecessary harm. That's why we built our own MCP server with very limited functionality, but tailored to our case.

Testing

And now the promised anecdote. Though it is not as straightforward as with conventional programming, unit testing helped us develop the various LangGraph nodes. Since a lot of the logic and fine-tuning was hidden in our LLM agents and their prompts, we prepared a mocked MCP server to let us test and add new features to these agents.

We developed the functionality with our tests, but then, once deployed to the test environment with real MCP servers, Data Gatherer sometimes did strange things. Thanks to observability, we identified that the agent was sometimes using other MCP tools. The thing was that our mock server had only mocked the few tools we thought the agent would use. But in reality the agent sometimes used more tools, in a creative way (and not in a good way ;-)). This was then fixed by improving the instructions. Less is sometimes more - especially for agents.

Still, automated unit testing helped us a lot as we came up with more improvements. And it also helped us test different scenarios, like waiting for a response, etc.

Observability

We installed Langfuse so that we can troubleshoot things. It might be a little bit more heavyweight than other tools, but it is the gold standard and our colleague had a good experience with it.

We had a small bug where our MCP server was not returning the latest messages when the conversation was too long. So when users complained that the agent responded weirdly, Langfuse helped us with the investigation.

Another anecdote is that Langfuse started to fill up the disk space on our server. So we had to implement a small clean-up script.

Real life problems

As mentioned throughout the article, we had to deal with multiple real-life problems that come with running AI agents in production - even for our internal purposes.

However, the toughest problems are usually not the technical ones. The worst problem we had was "adoption". People can ignore AI bots more easily than they can ignore humans. So the lesson learned is that AI agents are more helpful when a human wants something from the system than the other way around - when the agent wants something from a human. But we learned a lot anyway.