Re: "Honestly, I've been down this exact road and..."

• I actually think the issue is less about the training data and more about how Claude tries to fill in the blanks when it gets bored. I went through this exact mess last year with a trading firm and learned the hard way that Claude Code loves making up environment variables. You might want to consider defining a custom skill that specifically loads a local schema.json. I would suggest being careful with the rules:if syntax tho, it almost always trips up on nested quotes. I personally stopped using the CLI tool for the heavy lifting and moved back to Microsoft Visual Studio Code with the GitLab Workflow Extension to catch errors instantly. I spent like eighty bucks on API calls before I realized it just needed a golden yaml file to copy from. Definitely ground it with a specific version number in your prompt... it helps a ton.

Building on the earlier suggestion, I'd say you gotta be super careful with how much freedom you give the CLI. If you're burning through credits like that, it's usually because the context window is getting messy with all those failed attempts. I would suggest creating a dedicated reference file and explicitly telling Claude to only use the syntax found there. Tell it 'If it's not in this file, it doesn't exist.' Also, watch out for those Anthropic Claude 3.5 Sonnet API costs when you're scaling to 40 services. I'd make sure to feed it a 'golden' example of a working job and force it to use that as a strict template. It's way cheaper to have it copy-paste and swap variables than letting it 'innovate' on your logic. Just double check those rules:if statements tho... it still loves to sneak in old only/except syntax when it gets lazy lol.

Honestly, I've been down this exact road and it's a nightmare when you're on a deadline for a client like that. In my experience, these models love to hallucinate old GitLab syntax because the internet is just full of outdated StackOverflow posts from 2018. Over the years, I've learned that Claude needs a very specific leash. I tried using it for a similar mass-migration of microservices last year and kept getting the same rubbish with the rules:if logic. It felt like I was spending more time debugging the AI than actually building the infra... I've found a few things that help keep it from going off the rails:

• I strictly feed it snippets from the latest official GitLab schema docs in the prompt itself, not just my own internal docs.
• I tell it to explicitly avoid any tags that aren't on my specific runner fleet.
• I make it write the script in small chunks instead of the whole file at once. The problem is usually that the model defaults to the most common patterns in its training data, which happen to be the old, broken ones. It's frustrating when you're trying to automate and end up fixing syntax errors for hours. It just feels like double work at a certain point. Before you dump more credits into it tho, are you trying to get it to generate one master template for all 40 services, or are you having it customize every single pipeline based on the specific language stack of each service? Also, have you tried setting up a custom system prompt that explicitly blacklists those specific deprecated tags you keep seeing?