Test-Driving GCP Duet AI: A Promising Tool That Isn't Quite There Yet

I'm currently teaching at the Strasbourg Chamber of Commerce on a topic that isn't directly related to AI, at least not on the surface: deploying applications in DevOps mode on the Google Cloud Platform... I say "on the surface" only, because the Google Cloud Platform (GCP for those in the know) is packed with generative AI APIs, machine learning tools, and more.

Recently, a new product appeared on GCP: Duet AI. The students I'm currently training will later be tasked with sysadmin work, application deployment, IAM management, VPC, LDAP, the whole program! Something became very clear to me as I watched them learn new concepts like Dockerizing application stacks, CI/CD pipelines, and cloud permission management: modern distributed systems are complex for newcomers, and even for seasoned DevOps practitioners. Could Duet AI help them?

What Is Duet AI?

Duet AI is Google's version of Copilot. It differs from its GitHub competitor in that, true to Google's long-standing strategy, it's integrated into the Google Cloud Platform "natively": you can invoke Duet AI on any page of the GCP UI to ask a question or perform a task.

The tool can also generate code in an IDE, just like Copilot, making it a multi-faceted tool that serves both developers in their editors and the "point-and-click" crowd on cloud platforms, as one of my colleagues (whose identity shall remain secret) likes to call them.

AI-Powered Assistance: A Good Idea for Newcomers or Not?

Today, I'm more comfortable with certain concepts, which means I can get by on multiple platforms (Azure, AWS, etc.) without being an expert. I won't pretend this didn't come at the cost of countless hours studying seemingly endless nested concepts. Back then, ChatGPT either didn't exist or hadn't reached its current level of performance. It would have been a tremendous help in my learning!

We recently had a discussion at Reboot Conseil about providing AI-powered code assistance tools like Copilot: some of us believe that a developer needs to go through those long nights reading obscure docs and making mistakes to acquire a certain maturity and the necessary reflexes. The argument holds perfectly, because the process is indeed highly formative. Others (namely, me) felt that new professionals should use these tools as early and as much as possible to maximize their productivity and be profitable for their company, and most importantly, to simply have fun! So I'm currently thinking about how to familiarize students with prompt engineering for problem-solving, documentation, and using LLMs as thinking assistance tools (a rubber duck on steroids, if you prefer). I'm going to test Duet AI to form an opinion before discussing it with them.

This article will recount my first hands-on experience.

Duet AI 101

To stay in line with my training, I go straight to the use-cases tab in the documentation and proceed to... try deploying something!

I'm offered Duet AI tutorials on:

• data analysis
• developing a Python app
• creating a private build environment
• developing a Node.js app
• architecting a multi-tier app
• creating and modifying a VPC network
• creating and modifying a Kubernetes cluster

Not bad at all, but being part of the AI squad, what I want is to:

• deploy a VM with a GPU
• clone a git repo inside it
• retrieve data from somewhere
• train an AI model on it
• produce a compiled model and store it elsewhere
• kill the VM so it doesn't cost me money

and I want to do all this in scripted mode, please

Let's ask it! I don't know if we'll accomplish all of this in one article (or whether we'll need several), but I really want to try the product with something I'd actually need to do and see where it can take me with this kind of task list.

First Things First: The Money

To use DuetAI, you need to pay for a license, which can be monthly or annual. I try the monthly license:

... $22.80 for a one-month license is a bit steep, but maybe the results will be worth it. Let's continue. Incidentally, the GitHub Copilot Business offering has comparable pricing.

What Does It Look Like?

In the GCP console (I'm staying here for now and not installing the VSCode plugin) there's a button in the top right to invoke the LLM, to the left of the cloud shell shortcut.

... it pops up a chat window, let's get started with my project:

Here's my prompt:

I'd like to:

- deploy a VM which has access to a GPU
- clone some repo in it
- get some training data (probably an open-source dataset)
- train a regression or classification model based on this data - store the compiled model in a bucket
- kill the VM

I want to do all this in a CI/CD pipeline

The LLM Problem: Plausible But Inaccurate Answers

... and at first glance, it looks pretty cool! I'll paste Duet AI's very detailed response here; small caveat: it doesn't mention the CI/CD pipeline, so that will be the subject of a follow-up prompt:

To deploy a VM with access to a GPU, you can use the following steps: Go to the Google Cloud console . Click the Compute Engine menu. Click the Create Instance button. In the Name field, enter a name for your instance. In the Region field, select a region that supports GPUs. In the Machine type field, select a machine type that includes a GPU. In the Boot disk section, click the Change button. In the Public images tab, search for "Ubuntu". Select the latest Ubuntu image. Click the Select button. In the Advanced options section, click the Networking tab. In the Network tags field, enter allow-ssh . Click the Create button. Once your instance has been created, you can connect to it using SSH. To do this, you can use the following steps: Go to the Google Cloud console . Click the Compute Engine menu. Click the VM instances tab. Find your instance in the list and click the SSH button. Once you have connected to your instance, you can clone a repository from GitHub. To do this, you can use the following steps: Open a terminal window. Type the following command: git clone https://github.com/GoogleCloudPlatform/cloud-samples-data.git Press Enter. The repository has been cloned to your instance. You can now get some training data from the repository. To do this, you can use the following steps: Open a terminal window. Navigate to the cloud-samples-data/ai-platform/census directory. Type the following command: gsutil cp gs://cloud-samples-data/ai-platform/census/adult.data . Press Enter. The training data has been downloaded to your instance. You can now train a regression or classification model based on this data. To do this, you can use the following steps: Open a terminal window. Navigate to the cloud-samples-data/ai-platform/census directory. Type the following command: python3 train.py Press Enter. The model will be trained and saved to a file called model.joblib . You can now store the compiled model in a bucket. To do this, you can use the following steps: Go to the Google Cloud console . Click the Cloud Storage menu. Click the Create bucket button. In the Name field, enter a name for your bucket. In the Location field, select a location for your bucket. Click the Create button. Once your bucket has been created, you can upload the compiled model to it. To do this, you can use the following steps: Go to the Google Cloud console . Click the Cloud Storage menu. Click the name of your bucket. Click the Upload files button. Select the model.joblib file. Click the Open button. The compiled model has been uploaded to your bucket. You can now kill the VM. To do this, you can use the following steps: Go to the Google Cloud console . Click the Compute Engine menu. Click the VM instances tab. Find your instance in the list and click the Stop button. The VM has been killed. I hope this helps!

Duet AI even gives me a sample project to test the whole process! I read through the steps carefully, everything seems relevant at first glance... but is the linked repo in the response actually valid?

There is indeed a GoogleCloudPlatform account on GitHub with repos containing training data, but here the LLM either hallucinated a repo that doesn't exist or linked to one that no longer exists.

What about the command gsutil cp gs://cloud-samples-data/ai-platform/census/adult.data . for downloading data from a bucket?

... same thing, the link is invalid.

OK, let's continue our fact-checking: can I create a VM and clone a repo without any intermediate steps, as the response seems to suggest?

... yes, git is installed by default and I can clone repos.

Overall, the other steps in Duet AI's response are relevant, but they aren't really "actionable" because it's ultimately a detailed response repeating the sequence of steps I already had in mind from the start...

So I'm left wanting more. When I give the same prompt to GPT-4, I get detailed snippets and can boost my productivity without having to spend too much time in the docs:

... and when I ask Gemini the same thing, the LLM points me toward a better solution: Vertex AI ML pipelines.

Console-side conclusion: if you use Duet AI, my colleagues were right! A learner risks losing more time using the tool than by self-studying the documentation and trying to understand what they're doing.

Testing the IDE Integration

At this point, the Duet AI product is not satisfactory after this quick test drive... I was expecting quality code generation, or even semi-assisted resource creation to accomplish these tasks and move faster. In the end, I could spend less time reading the (admittedly quite clear) GCP documentation and/or prompting GPT-4/Gemini with more targeted prompts to handle a specific sub-step X or Y.

Now let's test the IDE integration (VSCode), maybe we'll get a pleasant surprise?

I'll ask it to write the code for a GitHub Actions pipeline matching my previous prompt:

Write the GitHub actions pipeline to:

- deploy a VM which has access to a GPU
- clone some repo in it
- get some training data (probably an open-source dataset)
- train a regression or classification model based on this data - store the compiled model in a bucket
- kill the VM

Here's what I got in response:

# This GitHub Actions pipeline will deploy a VM which has access to a GPU, clone a repo in it, get some training data (probably an open-source dataset), train a regression or classification model based on this data, store the compiled model in a bucket, and kill the VM.

- name: Deploy VM
uses: GoogleCloudPlatform/cloud-sdk-deploy-action@v1
with:
project_id: <walkthrough-project-id/>
region: us-central1
machine_type: n1-standard-4
image_family: tensorflow-2.12.0-gpu
image_project: tensorflow-cloud
service_account: <walkthrough-project-number/>-compute@developer.gserviceaccount.com

- name: Clone repo
uses: actions/checkout@v3

- name: Get training data
run: |
git clone https://github.com/GoogleCloudPlatform/cloud-samples-data.git
cd cloud-samples-data/ai-platform/mnist
gsutil cp gs://cloud-samples-data/ai-platform/mnist/train.csv .

- name: Train model
run: |
python3 mnist_train.py

- name: Store compiled model
uses: GoogleCloudPlatform/cloud-sdk-deploy-action@v1
with:
project_id: <walkthrough-project-id/>
region: us-central1
location: gs://<walkthrough-project-id/>-bucket/
source: .

- name: Kill VM
uses: GoogleCloudPlatform/cloud-sdk-deploy-action@v1
with:
project_id: <walkthrough-project-id/>
region: us-central1
machine_type: n1-standard-4
image_family: tensorflow-2.12.0-gpu
image_project: tensorflow-cloud
service_account: <walkthrough-project-number/>-compute@developer.gserviceaccount.com
delete: true

... the generation is super fast, but this is even worse: the GitHub Actions mentioned don't exist, the syntax is incorrect, it's really just a rough approximation of what a pipeline might look like on this platform!

Conclusion

I've come to the conclusion that this product is not yet mature, and it will certainly be worth revisiting later. I'm sticking with my GPT-4/Gemini duo and I'll add: using a single LLM, however well integrated it may be (like Copilot for the Microsoft suite), is not the future in my view. We'll always be limited by the inherent weaknesses of LLMs, even if they diminish over time. In my opinion, the most promising solutions are those that have LLMs collaborate with each other (multi-agent solutions like LangChain), that have access to external tools, or that enable an LLM to use introspection methods for self-correction (like self-corrective RAG)... These agentic applications will, in my view, deliver far more impressive results than integrating a single LLM into a software suite or cloud platform.

Thanks for reading, and see you soon!