ChatGPT for GOV.UK

Like everyone else, we’re (cautiously) excited by the developments that are going on in the world of Natural Language Processing (NLP) at the moment, and the high profile that the field has been given by the arrival and popularity of ChatGPT.

We’ve been experimenting a lot with ChatGPT and other Large Language Models (LLMs). In this post we talk about a demonstration application that we have built which layers ChatGPT on top of the UK Government website GOV.UK, allowing you to query the website in natural language.

This post is about what we have built, why we did it, and how it performs better than vanilla ChatGPT.

tl;dr

Sign up at https://chat.mantisnlp.com and have a play. To improve the application, we will record the conversations, so please don’t share any sensitive information with the chat bot.

What have we built?

We’ve built a layer on top of ChatGPT that improves its performance when asking questions related to GOV.UK. You can ask the chat bot for answers to questions that you would normally need to search on GOV.UK. For example if we want to know how to get a UK driving license:

Asking our chat bot “How do I get a driving license?”

The chat bot will return the relevant information along with links to the pages where it has found the information. We can follow up with more questions that build on the first one.

Following up with the related question: “How old do I need to be?”

Why have we built this?

Over the last year or so, we’ve had several requests from companies who wanted a chat bot that can be used to query internal company documents in natural language.

Large companies especially, that have huge volumes of knowledge, want systems that can make that data more easily searchable and accessible to their employees.

Imagine for example that you could ask a chat bot, in natural language, specific questions about your company’s past projects, business processes, or corporate intranet and get answers back immediately as if you were messaging a friend.

We’ve used GOV.UK as a demonstration because the content is well known (at least in the UK), is well structured (and therefore easy to verify), and is published under the permissible Open Government License.

How is it better than ChatGPT?

If you’ve experimented with ChatGPT (or other large language models LLMs) you may have come up against some limitations. OpenAI are transparent about these, they state that outputs from ChatGPT can be both out of date, and untrustworthy (hallucinated). Obviously this makes it difficult to justify using a chat bot like ChatGPT in a production setting where accuracy is important.

We’ve addressed these problems by anchoring the responses of our chat bot to a current version of GOV.UK, so that it provides more up-to-date information, and reduces the likelihood that the chat bot responds with something that is not accurate. We also provide links to the source of the information that the chat bot provides.

We can see this in action if we ask ChatGPT and our demo the same question: “Am I eligible for the UK Winter Fuel Payment”.

If we ask this question of ChatGPT, we get the answer (truncated for brevity):

Asking ChatGPT about my eligibility for the “Winter Fuel Payment”

This sounds plausible, but is not correct. According to GOV.UK, (https://www.gov.uk/winter-fuel-payment):

If you were born before 26 September 1956 you could get between £250 and £600 to help you pay your heating bills. This is known as a ‘Winter Fuel Payment’.

If we ask the same question of our chat bot, satisfyingly, we get the correct answer:

Asking our chat bot about my eligibility for the “Winter Fuel Payment”

What next?

You can try our chat bot at https://chat.mantisnlp.com. You’ll need to request access, and then we will create you an account. Make sure that you allow us to send you updates when you sign up (and marketing if you are interested) so that you can stay up to date as we make changes to the demo.

We’ve also got a more technical blog post planned looking at what we have done in more detail, and some insights from early user tests. Keep an eye out for it over the next couple of weeks.

And of course, if you are interested in deploying something similar for your own documents, please reach out to us at hi@mantisnlp.com for a chat.

ChatGPT for GOV.UK was originally published in MantisNLP on Medium, where people are continuing the conversation by highlighting and responding to this story.