«It was a big and ambitious project. Our client serves more than 100 million users on their portal. We had to develop a complex classifier for the project. We understood that our work would impact tens of millions of people. And of course, it was a challenge for us. Unfortunately, we cannot name the customer - due to NDA conditions. But we definitely need to tell you about this project».
– Andrey Tatarinov, CEO of Epoch8.
The client faced a problem: users frequently reached out to the support chat, but there weren’t always enough operators available to provide a prompt response. Our chatbot was designed to step in and assist during these critical moments.
The Goal
Develop a system that serves as the first line of support.
Tasks
- Train the bot to recognize over 600 potential question topics.
- Enable dynamic dialogue management.
- Implement statistics and reporting systems.
- Provide nuances of business logic.
Process
1 - Analytics
At the first stage we focused on how the bot would identify question topics. To achieve this, we analyzed real operator dialogues from chat logs, developed an ML prototype, and trained a test classification model. This approach allowed us to confirm that the chosen technology stack was suitable for the client's tasks.
2 - Pilot Version
We built a miniature test version of chatbot with basic functionality, ready for demonstration. The client had the opportunity to interact with it. They tested it, provided feedback and approved the full integration of the system.
3 - Scenario Formalisation
At this stage, we tailored the chatbot's functionality to align with the specific nuances of the business logic. For example:
- If a client repeatedly asks questions on the same topic, the query is redirected to an operator.
- If operators are unavailable, the bot avoids attempting a redirection, even if the script typically requires it.
- if the internal system of the service fails, the bot responds with a predefined message addressing the issue.
- If the bot cannot fully comprehend a question but identifies its general subject, it asks a clarifying question.
«At some point, the customer's staff of operators ex. And we taught the bot to route users not to operators from the general pool, but to those operators who could answer specific questions. At the same time, the bot took into account the topics of all the user's recent questions in order to more accurately determine the group for redirection. In other words, the chatbot is able to adapt to new circumstances and conditions».
– Andrei Tatarinov, CEO of Epoch8.
4 - Intent Classification
Intent is a class or, simply put, a keyword. We organized these intents into a hierarchical structure, enabling the bot to understand not only specific queries like "recover lost passport" but also individual terms such as "passport."
Our ML model was arranged in a more complex way than in classical chatbots.
A classic classifier works like this: it has a flat list of intents, and it tries to determine which one the user's phrase belongs to.
In our case, the bot knows how to answer modified questions. If it understands the query immediately, it responds accordingly. If not, it identifies the general topic and continues working within the scope of that intent, improving the accuracy of its responses.
5 - Integration
Next, we prepared the technology stack for implementation and conducted integration testing with all components, including:
- The chat system
- The statistics collection system
- The customer's monitoring systems
After that, the chatbot was ready!
Stack
RASA
This is a powerful open-source framework for chatbot development. Other options were either too costly or lacked sufficient functionality, making this framework the ideal choice. Our tests and analysis confirmed that it fully met the customer’s requirements.
«RASA is a technical engine. It is parameterised with configs for scripting and training models. It has a technical interface for training, it has mechanisms for managing the lifecycle of models. It has an API, integrations with external systems: with Telegram, Skype, for example. You can extend it with your own components for connectors or for business logic elements».
– Andrey Tatarinov, CEO of Epoch8.
BOTFRONT
This is an interface that allows you to describe a dialogue in a visual editor. Our task was to create a user-friendly working environment for the client's copywriters. Botfront turned out to be the best choice - it's human-friendly, yet functional. With its help, you can easily analyse the dialogue and make changes to the bot's behaviour.
How we trained it
We upgraded the standard classifier to handle multiple levels of queries. First, it identifies the main category of a query—like "passport-related," "yes/no questions," or "requesting an operator." Then, it focuses only on the intents within that category. For example, if at the first step it became clear that the query was related to passport, the classifier ignores all 600 intents and narrows its analysis to the 10 intents related to passports, making the process faster and more accurate.
This approach enhances the bot's responsiveness. Scriptwriters can use intermediate groups to ask clarifying questions or implement other interaction logic, creating a smoother and more dynamic user experience.
We also completed our components:
- A manager for handling trained models.
- A custom classifier.
- Integration components for interacting with support systems.
RESULTS
- The customer now has a user-friendly web interface to define interaction scenarios between the bot and the system.
- It is now possible to upload training samples for questions and set targeted answers.
- A ‘Retrain’ button appeared in the interface: the customer can ask the bot new inputs by himself.
- The chatbot responds to queries in under 1 second.
- It supports several behavioural scenarios.
- The bot knows the answers to 600 topics of the service.
- The system can handle up to 10 requests per second.
The Team
- Andrey Tatarinov, CEO/CTO.
- Alexey Kanskikh, project manager.
- Anton Grechkin, teamlead.
