ChatGPT: How (Not) to Learn Programming

Intro

The month is November 2022. I was a sophomore computer science major in university, taking classes in Data Structures and Assembly Language, two classes infamous among students for arguably being the “weed-out” classes in the major. Alongside the fact that they’re both core class requirements where I went, if you couldn’t do well in either, there was no way you’d survive any higher-level class that followed.

Whether it was debugging how a pointer was causing problems, or merging arrays in ARM, I remember the struggle that came with these two classes all too well.

One Friday afternoon, I was sat in a student center working on a quiz on ARM procedures (effectively functions in any other high-level language). A question came up where I had to write a procedure to print a pyramid using # characters - that is, one # on top, three #s in the next row, and so on. I spent 30 minutes tossing things at the wall hoping something would stick, but I ended up throwing in the towel. I just went and submitted it, and finished a Philly Cheesesteak I ordered earlier from down the street.

It turned out only two students out of over a hundred got it correct.

The transformer that changed the world

The next month, I was just about to wrap up my studies for the semester, finishing my final few assignments. But just as exams were just about to start, something big was released to the public:

We’ve trained a model called ChatGPT which interacts in a conversational way. The dialogue format makes it possible for ChatGPT to answer followup questions, admit its mistakes, challenge incorrect premises, and reject inappropriate requests.¹

A chatbot that can actually give accurate and comprehensible responses to prompts in natural human writing? What the heck, sign me up.

At first, I gave it a few questions about data structure concepts I learned in the past few months to see how well it could explain everything. And behold, it explained them in a very well-spoken manner. Then I asked it to write some code implementing a binary search tree in C++, and it did so, flawlessly and near-instantaneously:²

It would even respond to comedic prompts really, really well (this is a ridiculous example but it proves my point):

explain bsts in pirate speak

— me

Arrr, matey! Let me spin ye a tale of the Binary Search Tree (BST) — a treasure map fer yer data!

🏴‍☠️ What Be a BST?

A Binary Search Tree be a tree where each node be a chest o’ treasure (or data). Each chest holds a shiny number, and the rule o’ the sea be simple:

The left branch o’ a chest holds smaller treasures. The right branch holds bigger treasures.

So, if ye be lookin’ fer a specific piece o’ booty, ye can just follow the branches, left fer smaller, right fer bigger, till ye find yer prize — or walk the plank if it ain’t there!

— GPT

It performed so well it didn’t even feel real. Where was this when I spent 6 hours struggling on a C++ hash maps project?! As I was about to transition into taking more difficult classes, like Algorithms and Operating Systems, I was incredibly glad a site like this one existed - it would make completing these types of assignments so much easier.

But at the same time, I was thinking the exact same thing as most of my peers, and to a greater extent many in the computer science community:

Would this be what kills software development?

Learning from GPT

It’s been over 2 years since ChatGPT was released to the masses, and I’ve definitely used it a fair amount for coding projects in that time, whether for academia or side projects. It really felt like I was truly unlocking my programming potential, able to learn just about anything on the spot at the press of the Enter key.

But as I look back on things, how much did it help me become a better programmer through not just increasing my productivity, but helping me learn computer science concepts?

Regurgitating

As I look back on my use of GPT, one of the areas I found it to be most successful in (i.e. providing correct or otherwise helpful answers) was in the area of summary. Ask it to explain any commonly-seen term or concept, from hash maps in data structures to the Apriori algorithm in data mining, and you’d be given its definition, logic, applications, and/or even an implementation in code, all in pretty simple terms:

In some instances, these ideas were able to click in my mind not after a professor explained them in lecture, but only after GPT described them in this manner. It was at this point I was able to go back to the lecture slides and read more into specific aspects of them.

Cutting out the mundane

It also does a great job serving as a glorified autocomplete. If I had a situation where I needed to write the same line of code over and over, or do a lengthy find-and-replace process, GPT - more specifically, GitHub Copilot integrated into Visual Studio Code - handled it amazingly:

AFCE = [Team("Buffalo Bills"), Team("Miami Dolphins"), Team("New England Patriots"), Team("New York Jets")]
AFCN = [Team("Baltimore Ravens"), Team("Cincinnatti Bengals"), Team("Cleveland Browns"), Team("Pittsburgh Steelers")]
AFCS = [Team("Houston Texans"), Team("Indianapolis Colts"), Team("Jacksonville Jaguars"), Team("Tennessee Titans")]
AFCW = [Team("Denver Broncos"), Team("Kansas City Chiefs"), Team("Las Vegas Raiders"), Team("Los Angeles Chargers")]

When writing this, I already knew both how to declare arrays and objects in Python, and all 32 NFL teams. I just needed to put all of them in arrays representing their divisions, and the most time-consuming part of that was manually going through every string and changing their values. Copilot suggested these changes near-instantly, allowing me to do that in a fraction of the time the former would have taken.

Performance on more specific problems

GPT began to fall short when I asked it to solve problems with more specific criteria.

About half a year after I really started to use GPT, I took an Artificial Intelligence class at university. For coding assignments, we had to use quite possibly the most unorthodox language I’ve used in the holy 2020s: Common Lisp. Not having been updated since the mid-1990s, and largely superceded by other widely-used languages such as Python, resources for coding projects in Lisp are unsurprisingly scarce on the modern internet. Maybe a few user-made libraries on GitHub with a couple hundred stars here and there, but nothing too well-established like Tensorflow or Scikit.

As the class ramped up, I inevitably ran into problems with Lisp, whether it was syntax or just getting my logic right. I asked GPT about these issues, and it did write code for me in Lisp that was syntactically correct, and justified how it achieved the desired functionality.

Very rarely did it ever actually provide code that functioned in the correct manner.

This experience established to me that GPT was far from as reliable as people hyping it up claimed it to be. And it’s not just limited to “obscure” languages like Lisp: even with languages like CSS and Python, if you provide GPT with specific project code and ask for a specific fix, the outcome is much the same. It actually gets worse the more code you provide at a time.

My main takeaway was that the less data it has available on a specific problem, the poorer quality its responses will be. You’d think this is an incredibly obvious thing one should know about AI and ML, yet you still hear all this talk on how it will replace software engineers and make it so much easier to cheat in an academic setting.

Educational “value”

I’m going to go back a few months now, closer to the time GPT released. I was taking an Algorithms class at college, which saw us learn different algorithmic problems and ways to solve them, including the notable NP-hard traveling salesman problem, the gateway to dynamic programming staircase problem, and even lesser known ones like Dijkstra’s Dutch national flag problem.

My friends and I all found it to be one of the more difficult computer science courses. There was a lot of out-of-the-box thinking involved; unlike in previous classes, solutions to these problems were not incredibly obvious at first, and often required applying common data structures such as hash maps, and algorithms like Kruskal’s, in seemingly unorthodox ways.

The course had a mix of coding and writing assignments. For the former, GPT of course was recently released at the time, and all the hype in the world revolved around that. So, what did I do with the high difficulty of many of the problems I faced?

Asked it for help to write my code, of course. I did ask it to explain its code and the concepts surrounding it, and I thought I was beginning to master the art of algorithms. But in retrospect, I hardly really “learned” the applications of these algorithms just by asking GPT tons of questions, no matter how much I tried to learn from its explanations. I found that the bulk of what I learned came from the struggles I had to face myself. And plenty of that came with the writing assignments.

Our professor gave us several assignments throughout the semester describing a problem solvable through algorithmic means. For example:

An articulation vertex of a graph G is a vertex whose deletion disconnects G. Let G be a graph with n vertices and m edges.

Consider the problem of finding a vertex of G that is not an articulation vertex - i.e., whose deletion does not disconnect G. Now, give an O(n+m) algorithm that finds a deletion order for all n vertices such that no deletion disconnects the graph.

The twist was that we wouldn’t solve it in Python. We’d describe our process in solving it in English.

The process involved a long series of steps that we had to write out, including:

• The type of the problem (optimization, decision, search)
• The target runtime (i.e. O(n log n) usually meant sorting, O(1) points towards a hashmap…)
• What general purpose algorithms can be used
• Any constraints, freedoms

After that, we had to give solving it a first try, not using resources, such as the internet or office hours, in doing so. Usually, this failed to produce a working solution, but it did aid me in thinking of how to begin directly approaching the problem.

Following a long break (typically 12 hours), I would try and solve it again, this time using any resources. However, I had written down enough to the point where I only needed minor corrections in my thinking (for example, mistakenly thinking Dijkstra’s algorithm ran in linear time), so I just needed a few Google searches to set my ship in the right direction.

It was from these writing assignments where I learned the most. Not only did it force me to become so much more familiar with the general-purpose algorithms, data structures, and runtimes, but I would end up using this approach in future endeavors such as internships and side projects, and it paid great dividends. And most notable of all, my use of ChatGPT or other AI tools during this process was minimal.

Teaching in a GPT world

Having described my experiences with GPT, I overall find it fair to say that AI shouldn’t be feared in the ways we were told to think of it: as a career ender and glorified cheating tool. I really don’t think it’s the most evil thing out there and that it shouldn’t be used at all, because as I described, it performs certain functions incredibly well.

What I do fear, though, is how new generations of programmers will try and learn from it. My TA friends and I have both seen GPT’s effects on students we’ve helped out and graded.

Our computer science program has an incredibly fleshed-out curriculum for its introductory classes, notably the very first course CS majors take. Students use a web-based Python environment called BlockPy which gives them basic Python problems, which on the inside, have hundreds of unit tests AND conditions for returning very specific error messages in plain English, for example telling students they need to use a for loop to iterate over an array. It couldn’t possibly be any more beginner friendly.

And yet, my friends who TAed that class were seeing dictionaries and lambda functions when neither concept was taught in that class. Year after year after year, there always seemed to be a massive cheating “scandal” where hundreds of project submissions were flagged for plagarism.

Going back to the top of the post, I mentioned that if you didn’t succeed in the Data Structures course, you wouldn’t be able to make it past any higher-level course like Algorithms. Do you think a student would be able to make it anywhere close to that point if they can’t take the time to learn the most basic coding concepts, and have to resort to ChatGPT to do things for them?

I totally understand learning coding is like trying to speak a new language, only with the weirdest syntax imaginable. But if you’re serious about progressing your career in computer science, ideally you’d want to do what you can to maximize your learning in a very hand-holdy environment, because one thing’s for certain: it will not be hand-holdy at all going forward.

How to really learn

So with all of this said, what are some ways to better approach learning, with and without GPT, in a very different landscape where all this AI talk is near-constantly shoved in our faces?

The classic: Don’t blindly copy code

This advice has been propelled around for years, even before ChatGPT, but it holds even more relevance now given not only the much wider breadth of problems generative AI can help solve, but just how accessible it is. Type in effectively a Google search query and BOOM, you have an answer right at your fingertips.

If you just go into autopilot and copy code off of a site like Stack Overflow, sure the code may work perfectly out of the box. But would you be able to explain what it does exactly, and how it works?

This becomes a massive concern in two cases:

• If you ever need to revisit that code in the future. Doing a refactor of an entire codebase, or need to revamp larger sections of it to implement a major feature? If you don’t know how that one block of coding jargon works, I wish you the best of luck proceeding with that ambition.

• If the code in question is flawed. During research conducted by professors Ashkan Sami, Foutse Khmoh, and Gias Uddin, the team found 69 code snippets on Stack Overflow that they could state “with some certainty… they are vulnerable.” The most common flaws included not checking return values, which in C++ could lead to a null pointer dereference causing a program crash, and insufficient input validation which could result in input being executed as arbitrary code.

  • Unrelated to the research, but in the worst case, malicious actors will just outright manipulate beginner coders into executing dangerous code. A (now deleted) question asked why a Python project wouldn’t open, pasting in code that attempted to execute… a base64 encoded string? Which when decoded, downloaded and executed another payload.³

  • This problem only became worse as code given in these threads were used more and more, enabling them to spread around coding forums quickly, potentially putting many more developers at risk of implementing exploitable code. In fact, ChatGPT is trained on Stack Overflow posts, too, rightfully heightening this concern.

So, what can you do? Why of course, understand the code you’re copying. Look at the response containing the code in full, which if written well, will explain its functionality and potential limitations. Or even better, do extra research yourself. This brings me to my next point…

Read the docs

Every language and library WILL have documentation. How else would projects be built using them? And, more relevant to GPT, how would it know how to provide users with functioning code using them? It’s all because their developers wrote detailed documentation regarding their features and functionalities.

I’ll use the reference for the Python library pandas’s loc function as an example. Here, it describes exactly how the function works - it returns a group of rows and columns given a label or boolean array. It describes all the possible inputs you can supply and their typings, as well as possible errors. It even provides numerous use cases that wouldn’t be obvious from the start, such as setting many values if they match a list of labels:

>>> df.loc[['viper', 'sidewinder'], ['shield']] = 50
>>> df
            max_speed  shield
cobra               1       2
viper               4      50
sidewinder          7      50

I started to take on this mentality about a year ago, beginning to realize that by using GPT, I might write code faster, but I won’t actually learn much from it. Having applied it for some time, I now feel much more prepared to give a confident answer as to how I’d approach a coding problem I’m given, using the functions under my belt.

In fact, through looking at the documentation, you may discover something new and useful that you never thought you needed to know!

For something even better, if you use a modern editor like Visual Studio Code or the JetBrains series of products, hovering over a function, datatype, etc. will give a tooltip showing the documentation entry right there:

Reduce not your thinking, but your boredom

There was a quote about AI I found online sometime back:

I don’t want AI to do art and writing so I can do more work, I want AI to do my work so I can do more art and writing.

While it doesn’t talk about software engineering in specific, I thought about it, and found that it definitely can be applied to it.

You can make AI do all your coding, resulting in you having to do an unholy amount of prompt-writing and debugging to get a barely-functioning and hardly-readable codebase.

Or, you can tackle the problems yourself and grow as a developer and problem-solver, and relegate AI to the mundane, boring tasks like line completion or find-and-replace to save a good amount of time, as I demonstrated in the Cutting out the mundane section.

Talk to others!

Seriously. It sounds so stupidly simple, but it works so stupidly well.

If you’re a student, go to office hours. You effectively pay for access to the professor and the TAs! Out of those directly in your network, they are some of the most qualified people you can talk to in order to seek one-on-one help regarding a project or a concept.

There is also a lot of networking value to be had from this. As you go to office hours more and more often, you’ll get to know your professors and TAs more, and they’ll slowly form into valuable connections to have. They’ll be able to vouch for you and your initative to seek assistance when you know you need it, and of your growth as a computer scientist.

You can even just talk with other people in your class about it - form some friendships and study circles to reason through difficult concepts together. Perhaps you’ll expand your network even further as a result, and even form lifelong relationships that will last far beyond the college campus.

Final words

In retrospect, the way I learned computer science concepts the most was through the struggle of figuring things out myself, the research I did on my own online through Google and documentation, collaboration with peers, and figuring things out with professors. Of course, spending many hours on end debugging a project is not anywhere close to my favorite activity, but I still came out of those experiences a more knowledgeable and confident programmer.

The revelation of ChatGPT and other AI chatbots was a massive one, advertising itself as a completely new way of researching and solving problems. It definitely works well as a less convoluted Google and autocomplete, but for more specific problems, if it doesn’t fail in solving them, you don’t benefit from your efforts merely prompting it. In fact, you can end up spending even more time or energy solving the extra problems that using GPT causes you.

As the classic saying goes: “good things don’t come easy.” And from what I’ve learned, the seemingly easy option alone, in this case ChatGPT, may make you better at asking a chatbot to respond in a desired way, or regurgitate words given an exact prompt or question wording. But it will not directly make you a better programmer or problem-solver.