When Systems Go Awry
September 30, 2013
By Philip Nickerson
Troubleshooting. Is there a more loaded word? To the hatchery technician, it evokes fear of job loss – Have I caused this problem? To the hatchery manager it evokes fear of product loss – How much longer can we keep fish alive without working systems? To the hatchery owner it evokes fear of profits lost – What will be the cost of bringing in the tradesman, consultant etc.?
Land-based aquaculture systems are layered upon each other and consequently, are very complex. Those of us who have the privilege and responsibility of operating and maintaining them are forced to wear many hats – plumber, carpenter, electrician, refrigeration tech, mechanic, biologist, chemist, meteorologist, and oceanographer to name a few.
My goal as an aquaculture engineer and as the technical manager of two land- based sites is to create systems that increase the amount of time that biologists can be biologists, and fish technicians can care for the fish, and decrease the amount of time they need to wear other hats. To accomplish this I need to troubleshoot well. There is no one-size-fits-all troubleshooting advice I can give, but, after 10 years, my approach is to reflect on the possible systems that are connected to the problem and check them out one at a time.
A hatchery is made up of many systems and things can get complicated in a hurry. The good news is that 99.9% of these systems run flawlessly 99% of the time. For example, we recently installed a small oxygen generator in a cabinet in our electrical generator room for the purpose of supplying our ozone generator. The ozone generator feeds a foam fractionator via a mixing pump. An ORP controller turns the ozone generator on or off. Another pump moves water through the foam fractionator. Treating the water to the correct ORP requires countless systems to work together flawlessly 24/7/365 and for over five years now this is exactly what has been happening.
But what if the systems fail?
In the coldest part of February this past winter the broodstock tank temperatures were dropping. It seemed that the boilers were running but the water being supplied to the tank wasn’t being heated. Time for reflection. What systems are connected to the boilers? In order for the boilers to work properly, at least seven other systems have to be working. We needed to check the following:
- Fuel in the tank
- Air supply in the room
- Power to the boilers and circulators
- Process water through the trim exchangers
- Process water temperature controls
- Fresh water to the boiler circulation loop.
- The boilers internal systems (fuel pump, ignition circuits, temperature controls)
Numbers 1, 2, 3, 4 and 5 were easily confirmed as being in working order with a glance at gauges and indicators or by touch, so we moved on to 6 and 7.
A boiler circulation loop is supposed to be “100% recirc.” A pump is used to move water through the boiler to be heated. The water then passes through a trim exchanger or other heat exchange device where it heats the process water. From the heat exchanger, the boiler loop water returns to the boiler to be heated again.
Touching boiler circulation pipes is a good way to check that the system is working. In this case, the freshwater supply line from our domestic well water system to the boiler loop felt warm. It normally would be cool to the touch given that it is sourced from our well. Clearly, the well water system was allowing the boiler loop water to back feed out of the boilers into the well water system.
Turning on a tap in the lab revealed that indeed there was no pressure in the well water system. Time again for reflection. Historically, when the well water system loses pressure, the cause is either an empty well or a frozen line from the well to the building. Checking the well revealed it was dry. This was a relief. Trying to thaw a frozen water line or replace a well pump outside in February with the wind chill off the ocean is neither a pleasant nor a fast mission. And time mattered here. The broodstock tanks were going to keep cooling off until the boilers were up and running again.
Vital check lists
The following seven things are vital to the water level in our well:
• Wet or dry weather
• The fuel station next door
• The fishing supply store next door
• Three separate hatchery boiler loops
• Hatchery washrooms
• Hatchery cleaning stations
• Two kitchen sinks
The weather had not been dry so the well was refilling quickly. Starting at one end of the facility and checking the visible lines and systems through to the other end revealed that a boiler loop no longer in use was the culprit. That boiler loop had been replaced by a heat pump and was left in place to serve as redundancy. But without it running, it had succumbed to the freezing temperatures. A water pressure regulator had frozen (and corroded) enough to come apart and empty the whole well water system.
Shutting the valve and turning the well pump back on was the only fix needed to get things back up and running.
A more recent incident occurred the Friday afternoon before a long weekend. It cost a maintenance tech and me two Saturdays.
Our phone line was completely dead. No dial tone. No static. When the whole hatchery is backed up by an alarm system that uses the phone line to dial out and notify staff of any problems, a dead phone system is serious. Somebody is not going home until it’s fixed.
Reflection on this one turned up nothing that could have caused a dead phone line. But a walk through the hatchery turned up a pump running dry. During training a standard procedure was being followed with the exception of one step. The “turn the pump off” step. The pump being a couple hundred feet from the action means that it is possible to miss this step; shut off the discharge valve, and not notice. Whether from water hammer or from water stuck in the pump heating up, the plastic pipe threaded into the pump discharge had separated from the pump. If a geyser occurs in the pump room, but nobody is there to see it and there is a phone jack on the wall above the pump, the whole phone system can be knocked out.
Said plug was installed facing the floor (so water wouldn’t drip into it) but positioned just right to catch a full stream of water from this pump.
Disconnecting the wires to the jack restored the phone line instantly. Fortunately, everything else in the pump room still worked.
The pump in question was only used once per day in a batch process. An interim solution bought us some time and allowed us to go home Friday night. Saturday was used to re-pipe the discharge. All was well… until the next Saturday. Apparently the geyser had reached the aged pump motor. Its bearings were loudly and unquestionably failing and the pump then tripped the electrical overload switch to get our attention. Another Saturday afternoon later and we had a newly rebuilt pump and motor up and running.
Anyone responsible for keeping a hatchery running will very likely relate to these incidents. In both cases, most important was a careful, thoughtful approach to troubleshooting. Hopefully, sharing these experiences and our solutions to them can help others when similar situations arise.
If any HI readers have had similar experiences (and we know you do), Philip would love to hear about them: firstname.lastname@example.org