Large language models are evolving at an astonishing pace. For instance, according to CodeX, GPT-4 can pass the bar exam. As a long-time policy expert, this keeps me up at night. Will my work be replaced by a graphics card somewhere in Silicon Valley? As a policy expert whose work heavily relies on legal interpretation, the pertinent question is how effectively these advancements apply to the field of policy.
Let's run a little experiment to see if I am needlessly losing sleep. Based on five different tasks, I will test GPT-4, aware that the results will not be reproducible (an issue in itself) and that I am probably not the best prompt-writer. The goal is to determine if AI can handle at least some of my work at this point in time.
Let’s briefly summarise what a policy expert does for a client who is a manufacturer selling products in the EU. The process typically involves five key steps:
Identifying Relevant EU Rules – Review upcoming and existing legislation to determine which regulations apply to the client's products, operations, or markets.
Interpreting and Translating – Simplify complex legal language and translate it into practical requirements.
Creating Implementation Requirements – Offer guidance on adapting internal policies, procedures, or product strategies to meet legal requirements.
Drafting Documentation – Create policy briefs, internal guidance, and compliance roadmaps.
Shaping Industry Positions – When necessary, develop policy positions and advocate with regulators or industry associations.
For task number 1, I kindly asked GPT-4 if it could find me the applicable EU regulations for a washing machine, applicable in 2025. It came with a comprehensive list, but it forgot the rules for the individual components, such as the motor.
I asked GPT-4 to create a summary of a text from a public consultation about the Circular Economy Act. It added ‘Green Deal’ even though it is not mentioned in the text at all. It appears to have a mind of its own. Pass, but only if you are willing to spend more time checking than writing.
Okay, task number two: Can it clarify legal text? Let’s have a look at simplification. The EU Battery regulation, (EU) 2023/1542, article 8.1, starts with this one sentence of 122 words:
“From 18 August 2028 or 24 months after the date of entry into force of the delegated act referred to in the third subparagraph, whichever is the latest, industrial batteries with a capacity greater than 2 kWh, except those with exclusively external storage, electric vehicle batteries and SLI batteries that contain cobalt, lead, lithium or nickel in active materials, shall be accompanied by documentation containing information about the percentage share of cobalt, lithium or nickel that is present in active materials and that has been recovered from battery manufacturing waste or post-consumer waste, and the percentage share of lead that is present in the battery and that has been recovered from waste, for each battery model per year and per manufacturing plant.”
GPT-4 provides the following simplified version of the text:
Starting from 18 August 2028 (or two years after a related delegated act is published — whichever is later), industrial batteries over 2 kWh, EV batteries, and SLI batteries that contain cobalt, lead, lithium, or nickel must come with documentation.
This documentation must show:
The percentage of cobalt, lithium, or nickel in the battery’s active materials that comes from recycled waste (either manufacturing waste or used batteries).
The percentage of lead in the battery that comes from recycled waste.
This information must be provided per battery model, per year, and per factory.
It does this well. I then asked GPT-4 to translate it into Dutch, and it promptly (no pun intended) forgot the percentage of cobalt, lithium or nickel. Pass on the simplification, though.
Task number three is about implementation. I asked GPT to describe the design requirements for the removability of portable batteries in a laptop. I was surprised to see it explaining that the battery cannot be glued or welded and then detailed possible exemptions. Pass.
So, task four, implementation, is about reviewing the rules with engineers and creating the design requirements. While GPT-4 can draft sections of the text, these will need to be carefully examined by an expert—honestly, I might as well write them myself. Pass on the text writing.
Finally, task five, advocacy, is ultimately a human endeavour. AI can’t walk into a room, sense political tension, or choose the right person to engage with. It can’t judge tone in a tense negotiation. And it certainly doesn’t have the ability to stay silent if needed. Much of this work depends on trust, timing, and influence.
In conclusion, after conducting this very basic experiment, I found that GPT-4 can be helpful for summarizing legislative texts and translating policy into plain language. However, its effectiveness largely depends on whether you are already a policy expert and are willing to invest significant time in verifying the results.
To all policy experts who share my concerns, it’s essential to recognize that while you can utilize tools like GPT-4 to enhance your efficiency, you still need the intuition to determine what is relevant for your clients, what is feasible, the potential risks involved, and the best strategies to adopt. While GPT-4 can help accelerate your work behind the scenes, it cannot replace your judgment regarding how policy developments affect your clients. AI can draft your message, but it cannot build your reputation.