sia.hackernoon.com

After the appearance of the GitHub Copilot Agent, I decided to try it on my real-life ASP.NET8 project. I tried assisted coding and produced 5.300 lines of code, of which 95% was generated. There is a very, very big quality advancement in usability of this version compared to the version I tried 3 months ago.

1 Real-life project in VS2022

I am working on the development of C#/.NET8/ASP.NET application, which now has around 123.000 lines of code (SLOC), in Visual Studio 2022.

Environment is:

Visual Studio 2022, 17.14.5
GitHub Copilot (GHC). License Copilot Pro+
Agent mode, GPT-4o, GPT-4.1, Claude 3.7 Sonnet
C#/.NET8/ASP.NET/JavaScript/jQuery/HTML/CSS/Bootstrap/SQL-Server

2 Trying GitHub Copilot

2.1 March 2025

I first seriously tried to use GitHub Copilot (GHC) on my project in March 20025, and I detailed the results in the article [1]. I found it practically useless, except for short snippets.

2.2 June 2025

After the appearance of the GitHub Copilot Agent, I decided to try it again. It is described in [2]. [3], [4], [5]. It showed so far as a good tool for template code cloning and generation

2.3 Comparison

I was selfish; I was just interested in how useful this Gen-AI tool could be to my project work. I did not care if they called it “chat mode” or “agent mode”; I just wanted to see what it could do for me and my project.

3 Project work done

3.1 Work with GHC assistance

In my project, I used GHC assistance to add around 5.300 lines of code, out of which 95% is generated by GHC. (measured by Visual Studio Code Metrics tool).
All that was done in the new Agent mode
This was still a task carefully chosen for GHC, in the sense that it is template-based. It only needed to clone the code and adapt it from the Account entity to the Contract entity.
Since this was a template-based task, carefully chosen for GHC, I am still not sure about usability for general code generation for brand new methods/functions. Needs to be verified first.

3.2 Things I did manually

It looked it had problems with locating and creating Razor .cshtml files. So I created empty files with proper names in the proper location and let GHC fill the content
It looked like it could not add new strings to .resx files. But it created string names in the code. I added proper 4 language strings to .resx
For simple HTML buttons here and there, it is easier to just add it than to write a command prompt
Moved code/methods, reorganized a bit to my liking
Some sophisticated EF-LINQ expressions I modified manually. Easier than to explain it in the command prompt
GHC tends to omit C# comments, and I like my comments, so I added/copied them manually

3.3 Perspective of a programmer

I was selfish; I was just interested in how useful this Gin-AI tool could be to my project work. I did not care if they call it “chat mode” or “agent mode”, just what it can do for me
In this version of GHC, syntax errors are very rare, but there were a few logical errors.
It is a bit of a limitation of us humans to review all the changes if changes are spread across too many places. I refused all that, even if that may be OK. I make GHC do the work again
I do not see the programmer role disappearing, it is going more be on a higher level, leading and verifying the work of Gen-AI
I GO SLOW AND REVIEW every line of generated code. Slow in review, but altogether saves time later because of fewer defects.
So, basically, it is a Generate-Review-Drop-Generate-Review-Accept cycle as a work process. Reviewing is a burden on humans and requires significant effort, because you are reading someone else’s code, but I just do not want to accept anything into my repository that I haven’t read and understood.
It is a bit of a limitation of us humans to review changes. That GHC thing can generate 200 lines of valid code every 10 minutes, but I can not read and understand that code that fast. There is also human fatigue appearing, you get tired from reading and reviewing the generated code whole day.
It actually feels (on this smaller task, based on produced results) a bit like human coworker, does not figure it right away, so you need to explain details more, it does quality job, but it does not do exactly what you asked and does some things his own way, and sometimes is more clever that you think it will be.
Based on modifications GHC made to the code of my original template C#/cshtml 4 files, I was able to follow “how GHC is thinking”, and it does think like a good programmer.
It does feel as if you have a competent coworker who gets instructions from you, and you can delegate work to him/her.
Commands to GHC are in practically natural language, no “pseudo-science” prompt engineering needed, I spontaneously stopped using # for marking file names, and it all worked. I just reasonably explain like I would to some other developer in an email.

4 Weird behaviors of GHC – probably bugs

It seems there are still bugs in the product. Plain VS2022/GHC restarts from time to time are beneficial.

I was working on some plain HTML and was giving a task to GHC to add some JavaScript, and GHC thing was suddenly so stupid. For an hour, I was changing different LLMs, it was telling me it added JS to the code, and I did not see anything or just unfinished fragments. I was thinking, how come it can do so complicated tasks, and can’t add just 30 lines of JS? It took me an hour to figure out. It all disappeared when I restarted VS2022. It seems that automation in Agent mode was stuck somewhere; it was not reading or writing files. It looks like GHC keeps some tmp files where it remembers its work. Some file corruption somewhere. It started to work well after a restart.

5 Intellectual Property and Copyright

I noticed that in point of time, GHC Agent loaded 750 source files of my project. It was looking for something, trying to find something, or learning about the project. Is he learning from my code? So, if I work with GHC, all my code is Open Source?

For example, I am working on a specific Bank protocol ABCD. After working 2 weeks with GHC, it has practically loaded all my source files. Has he learned and memorized all that?

What if a month later, some other user, somewhere, asks GHC to implement a similar app with ABCD Banking protocol? That GHC thing can, in several minutes, create a whole app based on what it learned from my source code. And, it can, of course, easily make it unrecognizable/different from my source code.

Where are my legal “Intellectual Property” and “Copyright” rights? You get the idea.

Who can guarantee that those AI companies are not behaving like pirates? They already claim that used all the text on the Internet ([7]) and their systems are still “text hungry”.

6 Conclusion: Seeing is Believing

I was a great sceptic, with a good reason. Marketing on AI is very strong, and Rock-star CEOs tend to overpromise in order to sell their company's products and shares. An example is Tesla’s Autonomous drive promised in December 2015 ([6]).

I was very disappointed in the version of GHC from March 2025. It did not look practically usable. The number of syntax errors and hallucinated code was so big that it required huge manual rework. Also, there were many hallucinated properties and methods, meaning that GHC did not look into other files of the project, just working on one.

In contrast, the version of GHC from June 2025 is practically without syntax errors. Hallucinated methods and properties are all gone, meaning GHC now reads (a number of) project source files to resolve dependencies. Delivered generated code all compiles, but it can have logical errors. GHC is now quite usable for practical projects.

Progress in the quality of GitHub Copilot in 3 months from March 2025 to June 2025 is much more than we usually see for software products. If it continues to progress in quality like this…well, by the end of the year, we will have a very smart code assistant.

So, the version of GHC from June 2025 starts to look like what all those media personalities were talking about, a replacement for Junior/Intermediate level programmers ([8]). I am starting to see the basis for that prediction.