Why is it important for system engineers to collaborate?

A system engineer’s job is very diverse. Unlike software engineers, their tasks are not always explicitly defined. They have to do whatever it takes to keep the systems running, the customers happy, and the stakeholders updated. To that end, they need to have access to information, tools, and personnel spread across different teams. This is why it’s critical that system engineers collaborate with each other and the rest of the departments.

Responsibilities of a System Engineer

Among other things, a system engineer is responsible for:

• Delivering highly reliable systems: This requires ensuring that all components/applications have the necessary resources, and are communicating seamlessly with each other.
• Delivering highly available systems: This requires creating robust active-standby architectures, load-balancing policies, and auto-restart procedures.
• Delivering high-performing systems: This requires running stress tests on pre-production environments and finding the optimal configurations/benchmarks for applications.
• Delivering 2nd and 3rd level support: This requires receiving incident reports and complaints from the call centers and providing fixes, or escalating the issues to the appropriate team.
• Creating bug reports and change requests: This requires in-depth knowledge of all relevant applications.

Why do System Engineers Need to Collaborate with Other Teams?

To do their job effectively, system engineers need to have seamless collaboration with all relevant teams, at all times.

Translating customer desires into change requests:

As we saw above, system engineers act as the 2nd and 3rd lines of support. They receive inputs from call center agents regarding new customer desires and expectations.

It’s first their job to analyze these desires from the technical and feasibility point of views, and filter out any impractical ones.

Once they think that it’s worth pursuing, they try to convert the non-technical customer expectation into a technical change/feature request. This request is then raised with the developers and other relevant stakeholders.

Everyone involved needs to discuss the viability and implementation plan of the request, before it starts getting developed. This will only be possible if the tools used by system engineers (to create any change requests or lock any requirements) are collaborating with tools used by project managers and developers.

Creating level-3 bugs/change/feature requests

System engineers always have one eye on the network management system and the other on the requirement management tool. They are continuously looking for any errors/alerts occurring on any applications/systems, identifying their causes, and notifying relevant parties.

Let’s suppose an application crashes. Their course of action will be as follows:

1. Look at any logs that preceded the crash.
2. Check for any indicators of failure.
3. Identify the root cause.
4. Create a bug for the relevant development team. Attach the logs and any other helpful information with the ticket.

But their job doesn’t end there. They have to keep following-up with the relevant developer until the bug is resolved. Once again, this warrants the presence of collaboration between the tools used by both.

Evaluating scalability requirements and contributing to system design:

Nobody knows the deployment architecture and requirements better than the system engineer. Not even the application developer. Sometimes when new scalability requirements are raised, system engineers also evaluate the need for additional resources or changes in the system design.

Let’s suppose a company is about to increase its subscriber base from 500,000 to 5 million. They are currently using MySQL which does the job for a relatively small userbase, but might need some serious optimizations to handle millions of transactions per hour.

System engineers know how the traffic looks on a usual day, which might lead them to suggest upgrading to a more robust database solution like Teradata or Oracle, in order to handle the extra load.

To have discussions on system design and architecture, tools used by architects, developers, and system engineers need to collaborate.

Performing stress tests, calculating benchmarks, and suggesting improvements

Another way system engineers evaluate and improve the performance of applications is by performing stress tests. By bombarding application interfaces with bursts of requests, they calculate their breaking points, benchmarks, and performance results.

If applications are not performing up to the mark, they raise their concerns with the developers, along with any suggestions. This to-and-fro communication requires collaboration between different tools.

Today’s Situation: What happens when you don’t use collaborating tools?

Most organizations today don’t use collaborating tools. This leads to system engineers not having a holistic view of the system. They aren’t synced with the developers and other stakeholders.

1. Bugs in different states across different tools: A developer might have already resolved a bug in their issue tracking tool, but the system engineer has no way of knowing it, unless both the tools are collaborating.
2. Lack of traceability (different requirements across different tools): Requirements specified on the requirement management system by the system engineer might be different to the requirements on the change management tool. This often happens when requirements are communicated via unofficial channels like calls, IMs, and emails etc. However, if the tools were collaborating, it would be really easy for the developer to quickly correlate both sets of requirements before implementing the change.
3. Change in configuration settings without notifying the developer: If a system engineer notices a potential of improvement in the configuration/deployment settings, they might make the change without notifying the developer or other relevant people. This change might never get documented or get lost in informal communication channels. However, if there was a way to formally and quickly communicate the change to the developer (and seek their approval), they could have done so.
4. Development of a feature based on an informal discussion: With no way to formally lock the communication between a system engineer and software developer, it’s possible for features to get developed without any formal change requests. This will also mean that these features got developed without documented approval of relevant stakeholders. This can lead to lack of traceability and ownership of features.
5. Too much time wasted in meetings: As there is no official channel for all stakeholders to collaborate and discuss potential features/bugs/changes, a lot of time is spent in scheduling and conducting meetings.
6. Lack of communication = disaster/delayed error resolution: Let’s suppose a bug, known to the developers, causes a crash in the production environment. The system engineers, with no way to collaborate with the developers, try to conduct their own investigation in the beginning. They spend hours trying to find the root cause before escalating to the developer. The developer suggests an instant fix, as they were aware of the problem.

If there was an official channel to collaborate, the bug would have been fixed much sooner.

Real-world examples of lack of communication leading to tragic disasters are the Boeing 737 Max plane crashes. Boeing engineers introduced a new flight control component to these planes, namely the Maneuvering Characteristics Augmentation System (MCAS).

However, it’s reported that the pilots were not provided enough information about the MCAS, and how to respond in case it fails. This, along with the lack of redundancy in the sensory inputs of the system, are believed to be the main reasons for the crashes.

The Benefits of Using Collaborating Tools

Collaborating tools enhance the productivity and efficiency of their users. Here are some of the tangible benefits:

• Timely creation and tracking of level-3 bugs: A level-3 bug often has a higher priority than any new feature or change. They have to be resolved as soon as possible. Collaborating issue tracking, change management, and requirement management tools make it possible for system engineers to create bugs for developers and track their progress easily. Any updates/comments from the developer are also visible to them immediately, which decreases the mean-time-to-resolution immensely.
• Formal discussion between stakeholders from different teams: In case a new requirement has to be discussed, vetted, or verified by multiple people, collaborating tools provide a great medium to do so. This is especially important if a change has to be implemented urgently, but needs formal approval from various stakeholders.
• Everyone is on the same page: Be it a functional requirement or a non-functional one, collaborating tools ensure that everyone is on the same page about it. There is end-to-end traceability of a requirement at any given time. In case there is a mismatch/discrepancy in the requirement on different tools, it can be spotted before it’s too late.
• Discuss change impact, test cases etc.: Using collaborating tools, system engineers can discuss potential impacts of a change with developers on a formal channel. Additionally, they can also formally discuss the accepted test cases for an application/feature. This is especially important before deploying something on the production environment.