Copilot Exam Assistant... Epic Fail!

So ChatGPT, large language models, generative AI etc is so awesome, right 

Well, let me tell you a little story. Yesterday I had to renew two Microsoft certifications. Renewal exams are open book so you can search the internet to find the correct answer. I'll interpret that as, "you may use ChatGPT / Copilot / whatever to help with exam questions". They were the two "solution architect expert" certifications too that I really don't want to let expire. I won't say I was lazy, but I was feeling particularly unenthusiastic about having to use my brain on that day when I just wanted to get out of the house ASAP and go spend my time on something useful, such as... building a boat, for example. A brilliant idea popped into my head - hey, why don't I copy paste the exam questions into Copilot as fast as I can and smash through this renewal exam in like 15 minutes. I did just that. Now I won't say I'm proud of doing that cause it felt like the ultimate form of cheating, but fuck it, it's open book, and I know this shit, but let's say I was a bit pressed for time and really didn't want to waste it on some silly exam questions.

Aaaanyway, to cut a long story short: I FAILED THE EXAM! I was flabbergasted. How?! This is Microsoft Copilot, and it gave me the WRONG ANSWERS to some "simple" Microsoft renewal exam questions. How could that be when generative AI is so awesome and, you know, has all this intelligence, distilled knowledge and reasoning ability. I was mad because then I had to do a second attempt at passing the exam and I actually had to READ the questions and THINK about the correct answer. I passed both exams an hour later.

Going back to doing a bit of root cause analysis on the epic Copilot exam assistant failure, it boiled down to two things:

  1. If it gave me the wrong answer and I then prompted it with something like "why isn't option y the correct answer" the AI quickly changed its tune and responded to something like "you're right, I apologise for the confusion. Option y is the correct answer because such and such..." It was basically like arguing with a compulsive liar or a particularly spineless person who would just agree with you on whatever the moment you challenged them.
  2. The references it generated to support its answers were quite often just made up of hyperlinks that led to web pages that didn't exist.

So there you go. Next time you're feeling depressed and Google AI thingy tells you to consider jumping off a bridge, I recommend applying some critical thinking. Likewise, adding non-toxic glue to pasta sauce to make it stick to pasta and cheese is probably not the right answer.

Comments

Post a Comment

Popular posts from this blog

That pesky hard-coding

The other day I was reviewing a request for a quote and a statement of work for a project I've been working on for a while and my eye caught an interesting statement I haven't noticed before: ...the proposed approach of refactoring the codebase and procuring COTS products for replacement of hard-coded functionality is the lowest risk and most cost-effective approach to delivering an improved solution... Hard-coded functionality? Huh? What's that? Well, technically, all functionality implemented by means of software code is hard-coded in a sense or in other words, doesn't change unless you change the code logic. That's actually perfectly normal and good thing. You wouldn't want your software to go all fuzzy and produce different results each time given the same inputs. So clearly, whoever wrote the request for a quote didn't understand what hard-coded means. The term hard-coded is used in a slightly different context and is considered a negative pract
Read more

Poor Man's T-SQL Formatter

You know those nice little helpful add-ins and plug-ins that you discover one day and they really help with your productivity? Today I had to do some SQL scripting again and since I don't do it that often I get a little rusty. I also hate ugly looking code of any kind - anything that's not neat, tidy and sensibly formatted. Formatting is one of those things that's best automated - why would you waste your time meticulously making sure all your code is of consistent style, broken into new lines at the right point etc. Tools help and one such tool is the Poor Man's T-SQL Formatter: http://architectshack.com/PoorMansTSqlFormatter.ashx I discovered this one many years ago and I keep coming back to it because it is simply what I need, it works, it's quick to install and doesn't cost anything. Today I was working with Sql Server Management Studio 2017 and I wondered, hey does that plugin work with this version. Yes, yes it does. Awesome. Absolute kudos to th
Read more

Wingfoiling app for Garmin Watch

Image
Well, the other day, I got really frustrated with an app I've been using on my Garmin watch to track my wing foiling sessions. It kept crashing while I was on the water and I had to restart it several times when I noticed it crashed. I wasn't sure if the crashes meant the session recording data was completely or only partially lost. This is definitely not something I want to be worried about when I'm out on the water having fun wing foiling. I would also prefer to have my sessions recorded reliably as a single session so I can see how my skills are progressing and have reliable session statistics available in the Garmin Connect dashboard. This wasn't the first app I installed from the Garmin App Store either. Oh no, I think I installed and tried at least four, and some of them didn't even start or run to a point where I'd be able to test them. App crashing was a disappointingly common occurrence. When I came home that day, I decided to look into Garmin app devel
Read more