1 GitHub Copilot in VS2022
I am working on the development of .NET8/C#/ASP.NET8/EF8 application, which is now around 123.000 lines of code (SLOC), out of which 50.000 is EF-database-first model, in Visual Studio 2022.
I have a subscription to GitHub Copilot Pro + license. So far, that AI tool has been good for limited-scope tasks. I wanted to try the new GitHub Copilot Agent mode. Below are notes from my regular work.
Environment is:
- Visual Studio 2022, 17.14.5
- GitHub Copilot (GHC). License Copilot Pro+
- Agent mode, GPT-4o, GPT-4.1, Claude 3.7 Sonnet
2 Anecdotal Experience with Real ASP.NET8 project
The task is described in “Part2” of this series. Here is just my further analysis.
3 Analysis – Impressions
3.1 Human reviewing GHC-generated code in the previous task (regarding Part2)
I was looking very carefully at the generated code. Here are my findings.
- First, it is clean of syntax errors (C#) or similar nonsense, like extra brackets, which was a big problem 3 months ago.
- Not a single time I saw “hallucinated C# class property or member” that I saw 3 months ago in quantity that made generated code unusable because of re-work needed to fix all that garbage. This time is much, much smarter, I would say not a single level of quality up, but several levels of quality up. I do not know how, but that was a big problem, and it looked like GHC was trying to “guess out” properties names, instead of looking into the EF model. This time, it looked into other files of code of the project (no other way to find the right property names) and did the comparison/match. 3 months ago, I was thinking that LLMs are unusable for coding because of the hallucination problem, but they solved that. Even the fact that my DB table names are partly English, partly German, was not a problem; it figured that Contract=Vertrag, Account=Konto, etc. It is not that simple for a machine; English and German names are mixed in DB table names, C# class names, method parameters, etc.
- I told it to copy ALL COMMENTS, but it didn’t. It copied most of it. I am human and I need those comments, because 3 months later, when I look into the code, I do not remember proper API usage, so I leave comments on why and how it was invoked. I reviewed the code and copied the comments I like myself. It is not a big problem, but I noticed, GHC does not follow the given instructions strictly, it is “like a human coworker”, it does most of it, but changes a bit to its liking.
- Because the task was to make new code based on the original template, it looks simple. But it is not for the machine. So, GHC changes are right, it looks like it was able to “understand” the code and figure out where it needs to change the template, and where it doesn't. That is not simple at all, but it figured it right. There are many technologies involved, MVC Razor/HTML,CSS/JS/jQuery/Databable.net in 370 lines of code (.cshtml), and to figure out which string belongs to which library is no easy task for a machine. Simply, as far as I can see, GHC figured it all out and got it right.
- I didn’t ask for it, but during copying from the original template, it removed some dead JS code. I probably copied some JS functions, but later I did not need them, so they stayed there. OK, the new file is faster because it will need to load 15 lines of JS code less in the browser, but that is not the point. The point is, GHC “knows what is doing”, and that is sometimes what you want from a coworker, not just to do the task, but to fix little things-problems it notices during work, without explicit request for that.
- There is one subtle thing I am doing in my ASP.NET8 View-Component and GHC didn’t get it at first, and I needed to ask for several commands to finally do it, but once it figured it out, it figured it out better than I expected. So, in a view file (.cshtml) I am using a boolean flag ((Model?.IsSelectable), to make from one component actually 2 components, one editable and one read-only. Based on that flag, I show or hide some HTML/Datatables.net elements in .cshtml file. It is kind of tricky, because you have a mixture of MVC Razor/HTML/ CSS/JS/ jQuery/Databable.net languages. It looks like GHC didn’t get it in the first iteration/command. Then in the second, it copied HTML that is generated only in the default case Model?.IsSelectable==false. In the third iteration/command, I told it to include that flag, and it did with all the proper syntax. I still didn’t run it, but by reviewing the file visually, it looks like all the HTML/JS syntax is OK. But it did even more. I am using a boolean flag ((Model?.IsSelectable) in the original template just to show/hide visual HTML elements, and load ALL the JS code anyway. What GHC thing did is, it got the idea, why load all the JS code, when you do not need the “selectable part of JS code” if the component is read-only. And based on its own initiative, it implemented the same Boolean flag to include/exclude parts of JS code. Yes, I agree, that is an improvement, not big since that is maybe 100 lines of JS, but I did not care about that because I was just happy to get the component working. I do not know how GHC works from inside, but from my point of view, it appears that GHC “understands” that the .cshtml view file is about to be loaded into a browser, and for performance reasons, optimizations are possible. And on its own initiative, implemented optimization, and as far as I can see, looks like it all syntax and functionality is fine. It is a small performance improvement ( 100 lines of JS), but the big thing is “GHC looks like it knows what it is doing”.
4 Conclusion
Again, this was a small task, and this was still a task carefully chosen for GHC, in the sense that it is template-based and component-separated from the main code. It only needed to clone the code and adapt it from the Account entity to the Contract entity. But before, GHC would fail even in such “Gen-AI favorable scenarios”.
But, compared to several months ago, this “GitHub Copilot Agent” looks much, much smarter. It actually feels (on this smaller task, based on produced results) a bit like human coworker, does not figure it right away, so you need to explain details more, it does quality job, but it does not do exactly what you asked and does some things his own way, and sometimes is more clever that you think it will be.
Based on modifications GHC made to the code of my original template C#/cshtml 4 files, I was able to follow “how GHC is thinking”, and it does think like a good programmer.
To be continued…