Two-Phase Plan Mitigates Risk; Maximizes ROI
Challenge
Work in the Pulp & Paper industry is never without unexpected challenges, even with a great project team and a well-known customer site. This batch project was no exception. To ensure a successful project our engineers at Avid Solutions had to adapt to unplanned scheduling changes in such a way as to provide the customer with a safe, secure startup in order to ensure mutual success.
Solution
When the customer’s outage schedule changed unexpectedly due to maintenance issues, meeting the production schedule meant installing a new/modified batch management system while “online” without having to stop overall plant production. Simulations and extensive testing substantially mitigated the chance for error during this high-risk/high-reward strategy. Although this type of software testing increased upfront labor costs, it provided significant return on the investment. The result was a downtime of one batch cycle, zero wasted raw materials or product, and uninterrupted paper-machine production.
Background
The production of Coatings and Additives (C&A) in the papermaking industry is an inherently “batch-oriented” process. Interfacing with the other continuous parts of the plant presents unique controls and startup challenges. At a minimum, a well-designed system must be reliable enough to feed continuous production on the paper machine and yet flexible enough to produce a variety of different formulations accurately.
For this project, the customer had an existing Honeywell Experion Batch Management solution which successfully ran its three mixers and supporting equipment. But, plans for a new coating formulation required that the old system be updated. The change introduced a new ingredient, a new dispersing machine, several VFDs, and Ethernet/IP for device control. Both the batch management system and the graphics needed to be updated to accommodate these changes. Additionally, the new formulation would be much more sensitive to inaccuracies during batch creation. The old code was a direct conversion from the previous Honeywell TDC and CL code. It was neither reliable nor flexible enough to support consistent production of a higher-quality coating formulation.
The paper manufacturer contracted with Avid Solutions to rebuild the system in order to accommodate the new equipment.
Avid’s site familiarity and extensive experience in creating feature rich C&A solutions, allowed our contractors to work with operations to identify and incorporate many quality of life (QoL) features into the project.
Some were small changes, such as allowing batches to be produced and held in the mixers. This simple change to permissives effectively increased the buffer storage capacity by 25% and reduced the likelihood of causing a process disruption. Other changes included fundamental alterations to the structure and operation of the batch code. New branches were added to the code that would allow for the use of multiple dosing measurement methods and would introduce dynamic batch analysis to automatically adjust the system for errors due to slow valves, pipe lengths, and other forms of deadtime.
These types of fundamental changes and improvements can add significant QoL, but they are not without risk. Sufficient testing and troubleshooting are required to make startup and install a safe and smooth process.
To mitigate risk on this project, Avid scheduled a threeday Factory Acceptance Test (FAT) in addition to planning the outage in two separate phases. The installation phases would coincide with two previously scheduled plant outages. The first outage would include loading all of the new/modified software, while the second outage would necessitate starting up the new equipment. This scheduling would provide the benefit of being able to troubleshoot the updated software on existing equipment and then being able to apply the lessons learned to the new equipment startup. Staging the troubleshooting process in this manner would help to reduce the sources of potential problems.
But unfortunately, plans shifted quickly when the plant’s recovery boiler needed to be shut down for unexpected maintenance. This prompted the planned first outage to take place while the plant was already down, a week earlier than expected, in order to prevent the site from missing its production schedule. Due to this unexpected event, the installation schedule for the new software needed to be revised.
Approach
At this point, the change in plans resulted in several revised scheduling options for startup and install. The three major options included the following:
- Install the new software and start up the new equipment during a one-phase install as part of the second outage.
- Install phase one early during the maintenance outage
- Install phase one using the original schedule but require that the transition take place online.
To ensure mutual success, Avid worked with the customer to choose an option that would maximize safety and also meet production demands while minimizing labor cost and material waste. Because of the previously scheduled paper trial, Option #1 did not work for the customer. The trials could have been delayed but there was no reason to cause this disruption to the mill production schedule if other solutions were more favorable.
The consultants at Avid recommended against Option #2. Starting up during a maintenance outage was not a favorable option if it could be avoided. The startup had a higher rate of success and safety when all parties had a narrow focus on only the startup and no other maintenance work. The site’s size required a team that was willing to take on diverse roles. The controls staff, for example, supported the entire mill and the C&A operators frequently assisted other machine operators during abnormal situations. Attempting to install and tune new software while teams were busy with maintenance would likely have resulted in increased costs and a longer startup time. In addition, any test coating produced during the startup, successfully or not, would likely have been wasted.
With both offline options undesirable, the Avid team worked with the customer’s operations and project management teams to determine if an online startup was a possibility. Staying with the original timeline afforded the benefit of being able to complete the final scheduled testing, allowed a focused startup, and met the mill’s production plan. C&A, a batch process supplying a continuous production, theoretically supported the idea from a process design standpoint. However, installing software online always presents an inherent risk. Considering the total storage volume, formulations needed, and rates of consumption, the client estimated an install window of approximately four to five hours between producing the last needed batch on the old system and when a successful batch completed on the new system would be needed. Any longer than five hours would result in a machine shutdown causing the operational expenses to quickly spike to thousands of dollars per hour. In other words, Avid had five hours to complete the startup. No small request.
To go with this option, the consultants at Avid proved to the customer that the install could be done safely and within the time window. Several key features included in the Avid project proposal made this possible:
Testing: Part of the project included a three-day FAT. During the process, the two software designers from Avid worked with the customer’s project manager, controls engineer, production manager, an experienced operator, and an engineer from the equipment vendor. Involving so many people in the process represented a significant investment, but having everyone in one room proved to be very beneficial.
Because all the stakeholders were present at the FAT, they could make expedient decisions.
Simulation: To facilitate testing, Avid developed a lowfidelity, loop-back style simulation of the process. With the simulation in place, the operator’s experience with the graphics and batch functionality at the FAT very closely resembled the plant operation. In addition, it allowed Avid’s consultants to stress-test the software in as many simulated abnormal conditions as possible, eliminate all bugs, and ensure the system could always recover to a safe state.
Startup Planning: Confidence and safety under such severe time constraints required a plan. For this startup, Avid’s consultants developed a list of all the old code that needed to be deleted, the new code that needed to be installed, and a step-by-step list of the tasks to accomplish both of those goals.
Virtualization: Developing in a virtual environment allowed for a simulation of startup conditions. Avid’s consultants created another virtual machine with the existing code installed and walked through unloading and deleting the old code and then fully installing the new code. This test helped to find a bug in this particular version of the control system unknown to even Honeywell The solution required was not obvious. In this case, the import would have resulted in serious errors unless we changed specific system parameters during export. The virtualization proved to be invaluable. Had this error been found during an online startup on such a tight schedule, we likely would have had to abort the install and move to plan B.
A Backup Plan: Any good plan for risk management always includes a backup plan. For this install, Avid’s consultants decided to literally have a backup. The consultants decided that should something go wrong during startup, Avid could always reload the existing code and resume production. After another virtual pretest, we determined that reinstalling and loading could safely be completed in an hour. Comparing those results to the time window of reserve coating meant a safe install time limit of three hours. If the install wasn’t nearing completion at approximately the three-hour mark, the decision to abort would be made. This would allow ample time to reimport the backup and ensure that machine production remained uninterrupted.
Documentation: In addition to the usual IO List, this project also included a test plan, a test record and a software functional specification. The additional documentation walked a fine line between adding value and paying for binders that could potentially gather dust in a filing cabinet. However, these documents proved highly useful for mitigating risk. The test plan and record formalized what was accomplished during the FAT and provided sufficient record of what the team accomplished. Before and during a high-risk startup like this one, operators and engineers not present during the FAT invariably had questions about what had been tested. While the engineers trusted the team at Avid, the ability to point to a document with signatures from their managers and/or peers added an extra level of reassurance. In addition, having a functional specification document that described the changes and new features gave maintenance greater confidence in the ability to maintain the system going forward.
Conclusion
There are a variety of different ways to analyze the effectiveness of these efforts. In his Industrial Maintenance & Plant Operation article, Can System Integration Improve Your Triple Bottom Line? engineer and sustainability expert Mark Adair recommended evaluating the effects on the triple bottom line: people, planet, and profit.
People: Everyone involved in a project wants it to run smoothly and safely. Preserving the two-phase schedule for the startup had a variety of benefits to everyone involved. First, it helped to reduce stress and increase the morale of the project team. Second, it allowed ample time for operator training and QoL improvements. Installing software early during the first phase allowed the operators to be trained and gain confidence with the new graphics and features before adding in the complexity of new equipment. After years or and sometimes decades of operators using the same system, change can be difficult. Ample time for interaction with the operators also helped the Avid engineers to calm any anxieties about the new system and incorporate the operators’ valuable experience into the software. This led to a better working environment for those who are using the system day in and day out for years to come. Third, the two-step approach made planning easier. From a business perspective, the customer could meet the production schedule and start making use of the software’s new features. Avid was able to better plan our resource needs to ensure all of the needs of our customer were met. In addition, a set schedule meant fewer disruptions to personal time, whether vacations or time with family.
Planet: In his case study, Mark Adair reminds the reader that, “By reducing the occurrences of downtime, resources are conserved. Whether the plant is producing or not, water, steam, fuel and electricity are being used. These resources have their own environmental impact that is mitigated when the ent Because of extensive testing, an online startup was a viable option and this resulted in a smooth startup. Loading took about an hour and then the first batch of coating produced was within specifications and usable. Since the plant was online, operations could use this coating and didn’t have to drain it as waste. The result was a loss of zero raw materials during the first phase. The second phase of the install involved testing new equipment so some test batches and therefore waste were necessary. However, a multi-part installation improved accuracy and decreased waste here as well. The team could apply lessons learned from the first phase to the second phase, effectively increasing efficiency. This is particularly true when implementing newer technologies such as Ethernet/IP drives or unique equipment configurations. Testing those that were commissioned during phase one allowed the programmers to implement findings during the downtime between phases. When most the new equipment was brought online during phase two, fewer trial runs were needed, resulting in greatly reduced raw material waste.
Profit: Each of the features listed represents initial increased development and capital cost. However, a successful startup ultimately resulted in both cost reduction and cost prevention for the customer in the long term. The two-phase plan significantly mitigated the risk of interrupting or delaying operations, which could have been extremely costly.
This alone resulted in significant ROI. If something had gone wrong without a backup plan, the cost could exceed the entire integration cost in only a few hours. In addition, being able to maintain the schedule decreased contractor costs and delays. Good development, testing, and simulation allowed for no raw material waste. Loading software before and during construction allowed the team to commission equipment as installation was completed. In his paper Get Your Plant Back Online, On Time, consultant Tim Green suggests, “A healthy goal is to have 60% of the control devices powered and tested (minus logic) when construction is complete. You may even be able to perform water runs or simulated startups on portions of the project in order to do full, dynamic testing while construction and static check are being performed in other areas. This will further lessen the effects of delays induced by the construction team.” A startup time shorter than planned allowed the Avid team’s remaining time on site to be spent providing additional value for the customer by training operators, teaching maintenance techs, and continuing to add in other features. In all, these opportunities quickly exceeded the required investment.
Altogether, Avid’s consultants turned the negative possibility of a schedule change into increased benefits and value for the customer. The result was mutual success for both companies and project teams. Also, given this history and overall success, an online startup could be a preferential option for future companies.
