AI is revolutionising software development, making it more efficient, productive, and innovative. In this article, we explore the impact of AI in software development, delving into the insights gained from our investigation of GitHub Copilot performed by the ELEKS R&D team.
As the world of software development continues to evolve, the role of AI has become increasingly important. With benefits like the ability to reduce the time and effort required for coding and more, it's no wonder AI in software development is a hot topic nowadays.
92% of developers based in the U.S. are already using AI coding tools both at work and outside of it. - GitHub
70% of developers believe that AI coding tools will give them a distinct edge in their professional endeavours, including improved code quality, reduced completion time, and enhanced incident resolution capabilities. - GitHub
4 out of 5 developers expect AI coding tools to make their team more collaborative. - GitHub
Although AI hasn't reached a stage where it can independently create, test, and launch software products without human input, compared to the past, it has greatly speeded up the overall turnaround time.
Many AI-driven tools are available that can assist in developing custom applications by generating and autocompleting code, as well as detecting and fixing bugs. The ELEKS Research & Development team thoroughly assessed one such tool - GitHub Copilot - to evaluate its influence on developers' tasks, completion duration, and quality standards for recommendations provided. Let's take a closer look.
What is GitHub Copilot?
GitHub Copilot is a tool for code autocompletion and generation. It is built based on OpenAI Codex and trained on public GitHub repositories. As a result, it suggests code for any programming language. However, the quality of proposals depends on the number of public repositories that are based on a specific programming language and framework.
GitHub Copilot works with program code using Integrated Development Environment (IDE) plugins. Only the following IDEs are currently supported:
- Visual Studio Code
- Visual Studio
- Neovim
- all IDEs from JetBrains
Overview of ELEKS' GitHub Copilot investigation
This investigation aimed to explore the impact of using Copilot on both development speed and quality.
To conduct this study, a small pet project was initiated as a testing ground for various assumptions. With a team comprising four skilled middle developers armed with expertise in React, Redux, TypeScript, Jest, Vite, PHP, Symfony, and Codeception, we started the investigation and defined our goals and approaches as follows.
|
Goal |
Approach |
|---|---|
|
Evaluate the dependency between the developer’s competency and performance when using Copilot |
We selected the Symfony framework for back-end development. The back-end team had no substantial experience with this framework. |
|
Investigate the impact on typical developer tasks |
Before the project started, we created a WBS that covered the typical work of a developer on a commercial project. Also, we conducted a task estimation session. |
|
Investigate the impact on programming languages and frameworks by popularity |
We developed a web application with test coverage. |
|
Evaluate the impact on task completion time |
We made a preliminary estimation of individual tasks on the project. |
|
Enhance the research effectiveness |
We conducted knowledge-sharing sessions within the team during daily syncs. |
|
Explore the quality of suggestions |
We tested how Copilot deals with a non-classical and more complex project structure. |
Key findings from ELEKS' GitHub Copilot investigation
ELEKS' investigation of GitHub Copilot yielded several key findings that shed light on the tool's capabilities and benefits. Here is a more detailed description:
Please note that the investigation outcomes are based on the team's subjective feedback, observations of their work, and code reviews of their solutions.
The GitHub Copilot has two modes of operation:
- Code auto-completion mode – almost always positively affects performance.
- Code generation mode based on a comment with a prompt – works well only for typical code structures. It can be counterproductive when generating business logic or non-typical solutions because analysing the proposed code variants takes a lot of time.
The quality of GitHub Copilot code suggestions depends on the following:
- Popularity of the selected tech stack in public repositories. The more popular the technology or language, the higher the quality of GitHub Copilot suggestions.
- Code structure and application architecture. The more typical your solution is, the more effective suggestions GitHub Copilot provides. This is especially noticeable at the start of the project development.
Other advantages of GitHub Copilot:
- Over time, the quality of code suggestions increases. GitHub Copilot learns and adapts to the project structure, code style, and typical approaches to technical solutions on the project.
- Positively influences developer job satisfaction. GitHub Copilot takes on a portion of routine, typical, and boring tasks. The ratio between developing interesting and routine tasks changes, which increases developers’ comfort.
General team findings:
- The tool showed no impact on the code quality—neither positive nor negative. We think that for now, it cannot replace the code review process, manual code testing done by the developer, running autotests, security testing, and so on.
- The effectiveness depends on the developer's competence in the given tech stack. The better the developer's skills, the more effective this tool proves to be.
- The value of GitHub Copilot for learning new technologies is very doubtful.
- The team formed an impression that code suggestions become better if all code-related files are open in the IDE.
- Like any other tool, GitHub Copilot requires certain skills. The more the team practiced, the more positive feedback was received.
- The feedback regarding the increase in performance varies depending on the tech stack, the specialist’s competency level, and subjective reasons.
5–10% the average increase in productivity of team members. - ELEKS R&D team
On a side note, during the project, the team encountered several bugs in GitHub Copilot. For example, Copilot proposed code suggestions based on the code from another project open in another IDE window. It offered syntactically incorrect code. Copilot conflicts with the features of some plugins (built-in or added by a third party) that also have a code auto-completion functionality.
Conclusions
GitHub Copilot is a powerful tool that positively impacts development speed and developer job satisfaction. Like any other development tool, it requires time to master it.
At this point, an individual monthly license costs only $10, while the business plan is priced at $19 per month. GitHub Copilot will undoubtedly be financially beneficial for use on the projects, especially for the specialists of middle competency level and higher.
The team predicts that with the growth of experience using GitHub Copilot, the development speed can increase by 5–25%, depending on the technology's popularity and the developer's competency level. GitHub Copilot is guaranteed not to slow down development because developers will gradually gain the expertise and understanding of when it is beneficial to use the tool and when not.
In conclusion, the impact of AI in software development, exemplified by tools like GitHub Copilot, is undeniable. Overall, such tools as GitHub Copilot cannot replace a developer but serve as valuable tools to enhance developers' comfort and productivity, contributing to the evolution of software development in an increasingly AI-driven world.
Unlock the path to an efficient software development process and successful outcomes, contact ELEKS experts.
Also published here.