How not to get left behind by AI

How not to get left behind by AI: tips for getting started with a Pilot Project

The Economist recently published a piece entitled Your employer is (probably) unprepared for artificial intelligence (AI). It’s a thought provoking article that considers how technologies diffuse through an economy, and the negative consequences for firms of failing to keep up with technological advances.

Most striking for me were two quoted statistics: according to official statistics, only 1.6% of US firms made use of Machine Learning (an important sub-type of AI) in 2020; and: 40% of small businesses in the US are not interested in AI tools.

These are remarkable statistics given the benefits that AI is expected to bring. In June, Mckinsey published a report stating that modern Generative AI, like ChatGPT, could add as much as $4.4 trillion to the global economy. If that sounds like a lot, it is: for reference, the entire UK economy — the sixth largest in the world — was $3.1 trillion in 2021.

Hopefully this isn’t news to you, and your organisation has already got a comprehensive AI strategy in place, but if not, this blog post is for you.

Obselete technology. Photo by Sergey Svechnikov on Unsplash

It’s never been easier to get started

Thankfully, it’s never been easier for an organisation to get started with AI. One important way to do this is to execute a successful Pilot Project which can demonstrate the benefits to the organisation, and help to build momentum behind the idea of using AI.

Stanford Professor, and AI luminary Andrew Ng in his AI Transformation Playbook, advocates Pilot Projects as the first step for companies wishing to make an AI transformation.

In this blog post I’ll talk about what makes a good Pilot Project, and three simple rules for identifying one in your organisation.

What is a Pilot Project?

A Pilot Project is a project undertaken within an organisation to prove the value of AI. The idea is to choose a problem that is big enough that solving it would be important to the business, but not so big that demonstrable progress cannot be made within a 3–6 months. It should also be both technically feasible and have a measurable impact.

Now is not the time to shoot for the singularity, the objective is to get something important done, within the timescale, demonstrate value, and unlock budget for the next step in your organisation’s AI transformation.

There should be data

It may be necessary to train a new AI model during the pilot project, and data is essential to this process. Exactly what data means differs for every project, but if you want to show progress on a problem in 3–6 months, then the pilot project should tackle a problem for which some relevant data already exists, or can easily be produced.

It’s possible to generate synthetic data, or work with internal or external partners to annotate data, but it takes time to get these projects off the ground, and the fact that the data do not already exist may mean that the problem is not yet amenable to an AI approach in a short timescale.

The exception to this rule is that the new wave of Generative AI models such as OpenAI’s ChatGPT, Google’s Bard, and Anthropic’s Claude, are capable of doing few and even zero shot prediction. This means that you may need very little or no data at all to get started with these models. However, they may not perform well on problems that are very specific to your business, or are not related to natural language processing (NLP) which is the field of AI concerned with understanding and generating language

Moreover, even if you can use these so-called Large Language Models (LLMs), data may still be required to make a meaningful evaluation of the outputs, and there are trade-offs to working with these new models that need to be evaluated carefully, such as: high cost, lack of stability, and privacy concerns. And in the long run, a bespoke model trained on your organisation’s data is likely to perform better.

It should be easy to measure impact

At the end of the pilot project, you want to be in a position to demonstrate to your stakeholders that the project was successfully completed with a measurable impact. Measuring the impact of AI projects can be hard if we don’t have existing KPIs or metrics to act as a baseline against which we can evaluate performance. Let’s look at an example.

When I worked at the Government Digital Service (the part of the UK Government that manages the GOV.UK website) we did a pilot project to help a team there to understand feedback received from visitors to GOV.UK. There was already a team of people working on the problem, and we knew that it took a person in that team about forty-five minutes to process one hundred pieces of feedback. This meant that we could measure performance of our project in terms of hours saved per week for the team.

This is a good metric that was easy to understand, but it wasn’t a great metric because it didn’t speak to important business objectives of the organisation. What GDS really cared about was how to make content easier to find, and how to optimise the user experience, and our metric didn’t speak to these objectives at all.

Choose mosaics over monoliths

Another experience I had working in the UK Government is that our stakeholders from other Government departments often wanted to shoot for the moon, and try to build a monolithic solution that could solve all of their thorniest problems at once.

Aside from being a poor choice for a pilot project in terms of technical feasibility and timescales, the route to being an AI enabled organisation is not (yet) about building your own HAL9000. Instead, mature AI enabled organisations tend to incorporate AI as a mosaic of specialised tools and services that solve specific business problems.

This can be an important message to put across to stakeholders, and can help to manage expectations, especially in the early stages of an organisation’s AI transformation. It’s also important to remember that a pilot project is as much about the organisation learning how to execute these projects as it is about the project itself. AI projects often involve more uncertainty than traditional software engineering projects, and learning how to deal with this uncertainty is a key outcome in itself.

Who should execute the pilot project?

If you’re lucky enough to have internal capacity to take on your first few pilot projects, then that is ideal. It’s often the case however that organisations want to do a pilot project to prove out a potential use case for AI, before allocating the budget for building an internal team. In this case it makes sense to work with an agency on contract to get started.

You’re probably thinking that this is exactly what the founder of an AI consultancy would say, and you’re right, so let me restrict my advice to the following. It can be hard to select one agency over another when you don’t have specific expertise in the field, so take your time to select a partner who has the right experience for your problem.

I recommend spending some time in ‘discovery’ during which the agency will look to understand your problem better through a series of meetings with the key stakeholders, and looking at samples of your data. Ideally this discovery work would be discrete, and result in a report detailing the findings and options for a route forward. Should you decide that the agency isn’t the right fit for your organisation, you can use the report to scope work with another agency without having to start from scratch.

What’s next?

It’s a good idea to take on a few pilot projects to get a good understanding of the lifecycle of an AI project, but once enough momentum has been built within the organisation, you can progress on to the next phase of AI transformation. Andrew Ng’s recommendation is to build internal AI capability, and this makes sense as you’ll want to start to lay out a long term strategy for the organisation with full awareness of the organisation’s goals and objectives for the coming year, and it may be easier to achieve this with an internal team.

To conclude

Pilot projects are a fantastic way to take your organisation’s first steps into the world of AI. Keep them small and measurable, and use it as an opportunity to learn how AI projects differ from the projects your organisation is used to delivering, and the skills and profiles needed to deliver them.

And of course, it goes without saying; if you are looking to complete a pilot project Mantis is here to help. Drop us a line at hi@mantisnlp.com.

How not to get left behind by AI was originally published in MantisNLP on Medium, where people are continuing the conversation by highlighting and responding to this story.