Dealing with Software Development is not the cleanest and easy thing. It encompasses a myriad of issues, technical and non-technical. The technical aspect is surely much more complex and needs hard skills, but at the same time can be managed in some way, using the right tactics and tools. The non-technical world on the other hand has much more degrees of freedom. It has the same variability and unpredictability as human nature.

As with any other product manufacturing process, a number of best practices have been put in place and tested since the early years of the software development adventure. The agile methodologies are a set of different ways to cope mainly with extreme variability of the requirements, and also the lack of focus on the most important goals to deliver the final product.

Sometimes such methodologies go beyond their original purpose and the net result is far from ideal. Scrum is one of these methodologies. It's described more as a framework. It is based on a basic set of principles, rules, events, and roles. We discuss in this article how this framework can be used with judgment and flexibility and avoid some strict interpretations that could be far from ideal.

Very Briefly Introduction to Scrum

The Agile methodologies are based on the following general principles, defined in the so-called "Agile Manifesto":

(Source: Agile Manifesto)

According to the Agile Manifesto, in the above statements, while the items on the right are important, those on the left are more.

From the standpoint of the development process, the agile methodologies all imply an iterative and incremental process instead of the classic Waterfall model: the whole development is made of a number of incremental steps, and each step is made of the same phases that characterize a whole waterfall project: a gathering of requirements, analysis, design, and coding. That is to say, each step represents an increment in the whole development and can be seen as a whole project by itself.

The Scrum methodology can be seen as a particular declination of the above principles. Scrum is a framework within which a team can use its own processes to develop some specific software product. It is characterized basically by Roles, Events, and Artifacts.

The Roles are:

Events are:

Artifacts are:

Don't Confuse Scrum with Some Form of Doctrine

You can consider the above-listed items as a base by which a set of professionals can operate. They can be seen as a useful tool but what really brings cohesion, intercommunication, and effectiveness in a project are the people themselves. The fact is, even with all those prescriptions strictly followed, and all the commitment by the Scrum Master, a project can drift into chaos and fail miserably.

This is because people often confuse rules, methodologies, and frameworks with some sort of divine engine supposed to drive humans to the right path. It is a common psychological flaw. A very important consideration is that reality is different from theories, and it is a very complex and wild animal. If you think you can tame it with a list of rules and patterns you are doomed to failure.

The real purpose of Scrum is to minimize the amount of "background noise" in the development process and improve the focus on the most important things. If used with the right flexibility and modesty, it can be effective in doing so.

In the following section, I describe a real use case, in which I took a journey into the world of Scrum. It was a journey of inexperienced people, including myself, willing to embrace the agile principles in a consistent way for the first time. Many projects in which I worked before were made in an iterative way but without the whole ceremony of a real agile framework.

A Real Use Case

We talk here about a real use case, in which a far from strict adoption of the Scrum framework had been made. The project was about a software system aimed at tracing all the activities involved in an anatomical pathology laboratory, from gathering the tissue specimens, and treating them in various steps until the final distribution of the tissue slides. The system was also supposed to be integrated in specific phases with external automation machinery, by specific software drivers.

I was involved in this project as a software architect. I had to cooperate with the project manager in order to outline the main issues and devise a basic architecture blueprint. If you follow the Agile principles to the extreme, thinking of architecture beforehand is not the best thing. Even the architectural design is seen in an iterative scenario. In most situations though, you still need a base from which to start, and you have to discuss it with all the stakeholders involved.

I approached this preliminary architectural study trying to have a clear separation of the contexts, in a structured approach based on views, viewpoints, and perspectives. Following such an approach is important in order to have a clear separation of problems, and also to customize the discussion based on the particular type of stakeholders.

A part of the discussed architecture was indeed the development phase and its organization. The company itself had not yet adopted any agile methodologies. Nevertheless, in agreement with the manager and the other people involved, we wanted to fill this gap and we chose to get inspiration from the Scrum methodology framework.

We were not trained at all in Scrum, but we were all conscious of its basics. The main directives we had in mind for the project, both methodological and technical, were:

Scrum Practices

Being motivated by the need to enforce some solid methodology framework but forced by our lack of deep skills, we chose to take some of the main rules of Scrum without going too far. First of all an iterative approach was really important in our mind. Scrum covers this through the so-called Sprints, which we can consider the iterative units of work. Each Sprint is supposed to cover a chosen number of features from the backlog and has a specific time duration.

We chose two weeks for the time duration of our sprints. Before starting the real deal, we set up a conventional Sprint number zero of one week to do preliminary work and organizational tasks. In this preliminary Sprint, we also wrote the initial version of the backlog, with all the features listed by priority. We did not embrace a particular method for evaluating task efforts, just a normal discussion between team members.

At the beginning of each Sprint, as for the Scrum rules, we would discuss the work already done, assess all the encountered issues, and choose the features to be implemented in the coming Sprint.

The project manager played the role of the Product Owner, backed up by a domain expert. This made sense in the specific situation because there was no real product manager involved, and the project manager himself had all the knowledge of the client's requirements. As for the Scrum Master, I did that for a while and then pass the role to another colleague, even if both of us were not fully trained in it.

Our team was a heterogeneous one with people working from different cities. We were forced then to organize a virtual version of stand-up meetings by scheduling Skype calls every day at the same hour. The meetings were about 15 minutes long. Some of the team members has some form of resistance regarded this, maybe because they interpreted it as a form of control, which is not.

We spent some time making them aware of the real meaning of the daily meetings: a brief discussion between teammates to focus on the main issues, enhance communication and find ways to improve and make the work easier for all. For a while, they kept saying it was a waste of time and a source of distraction from the real work but in the end, we managed to convince them.

Tracking activities/sources by automation tools

Besides the methodology practices, we also needed the tools to solve these main concerns:

The flow of information in a project is far too complex to rely only on methodology practices to control it. You need a solid ground of tools to make the work easier. The most tasks you automate, the more you can concentrate on important things and produce a better product.

For code versioning, we used Git as it is the most natural choice. Git can be used in a variety of ways, and we personally adopted Gitflow as a workflow pattern.

To track the activities we used Redmine, which is a general platform aimed at project management.

To deal with the third concern, we integrated our Git repository with Redmine in such a way Git commits could be related to specific Redmine tickets by using an issue identification code in the commit comment. This way we had a full mapping between activities and Git units of work.

Behavior Driven Development approach coupled with unit and integration testing

We were strongly willing to base our development process on a Behavior Driven approach. BDD fits very well with Scrum and in general with Agile principles, especially in the part that regards communication. It allows writing test scenarios in a format that can be understood by nontechnical people, and with the right tools gives a visual report of the current status. It draws the logical boundaries of the product and forces the development work within those boundaries.

The BDD functional tests were just the external layer of the whole testing instrumentation. We used unit and integration tests as well. In order not to be overwhelmed by the complexity of such a development environment we had to use the right tools and automation features.

The main technologies involved were:

The following mental map shows a summary of the things discussed above:

Considerations

Was the adoption of Scrum, even if in a light and flexible way, worth it? The answer is yes. Let's be clear though, we cannot see it as the solution to all problems, and if adopted without understanding its very spirit could be even detrimental. The essence of a methodology is to offer a collective mind schema to make people work together with maximum sharing of information and commitment, but if the people are focused on following the rules of an exotic ceremony instead of the actual work, the original purpose vanishes.

A Sports Analogy

You can better understand what was said above with a sports analogy. Not all people like soccer, but almost everyone has at least a minimal idea of how it works. First of all, is a collective game. Two teams confront each other on a game pitch trying to deliver a ball inside a goal. To do this apparently simple task they have to mix individual initiatives with collective strategies and tactics. Those strategies and tactics are taught beforehand by the coach and make up a big percentage of the whole time spent in workout sessions.

To be really effective, the above-described collective rules followed by the soccer players must be executed without effort, and without even thinking they exist. They must be automatic, as the gestures to drive a car for instance. A basic requirement to reach this objective is that they have to be simple enough to be fully absorbed by all the players in the limited amount of time required to prepare for the championship.

If you are focused on following the Scrum ceremony instead of the real work you end up bringing chaos instead of order. On the other hand, if you follow a more pragmatic approach, being flexible and even getting rid of all the things that in our specific experience do not work, you can make the best of it. You can even think of postponing some Scrum approaches if you find them hard to follow, and then try to introduce them in a later phase.

Managing the Managers

Let's stress a concept: agile methodologies and Scrum specifically, can only work if the team people are either willing of adopting it or at least aware of its advantages. It cannot work if it is only introduced by managers in a company to follow the general fuss and tell the world outside: "See? We are agile!". Let's be clear: if the purpose is only marketing it would be better just to pretend of following those methodologies. There is no need to introduce in the internal processes a burden that has only a promotional purpose.

Moral of the story: agile methodologies can have some positive impact only if adopted by the engineers along with the managers and not just imposed by the managers. Most managers, especially those without a technical background, don't know anything about how a methodology could impact the life cycle of a software project, while engineers do, especially senior engineers. Years of experience are better than the best books and the best courses.

Furthermore, based on my weird experience in Italian companies, organizations are often dictated by a culture that seems to come from some sort of medieval heritage. In this context, the Scrum Master and even the Product Owner roles could be simply seen as a source of authority in a career path, rather than operative roles.

I have experimented with this harsh reality recently, to tell the truth. Normally these people do not have the slightest idea of what are the very principles of a methodology, and keep harassing people with rules they don't even understand.

In these horrible environments, Extreme Programming and Scrum are just meaningless titles. In these contexts the managers are sources of entropy to be dealt with, and to reach a decent level of productivity the team must manage its own work and even the managers, to reduce their bad influence. That explains the title of this section: "Managing the Managers".

Balance Between Methodology, Testing Practices, Tracking Activities, and Related Tools

In the use case described in this article, we have talked about methodology but also about testing strategies, tracking activities, and the tools used to support them. The best methodology frameworks cannot solve by themselves all the complex matters involved in software development.

As a matter of fact, BDD, which covers functional testing, is a very effective way of sharing knowledge of the logic of the software system being developed. It builds a strong bond between all the team members and the Product Owner, and it improves the focus on the more important aspects of the project. So, it covers part of the issues that Scrum is supposed to cover.

Unit and integration testing, on the other hand, are more involved in the inner processes of the software developers, but indirectly they make changes easier to deal with, coupling well with the iterative approach of Scrum.

Conclusion

Software development is a complex matter even in minimal projects handled by a single developer. If a project needs a team to be developed, a number of organizational issues arise. Agile methodologies are an attempt to solve those issues but they would be really useful only if taken if a grain of salt and by avoiding any form of senseless magnification.