Summary: Connectivity promises easier support, but every connection brings tradeoffs. Through four versions of the same Monday, a controls engineer discovers that getting a controller connected may be the easy part. Supporting the customer is what really matters.
There are a number of hot topics in industrial controls these days. The publications encourage conversations surrounding Determinism, Edge Computing, AI, Security, IT, Connectivity, and Enterprise Solutions. Most of that focuses on larger situations and manufacturing facilities. It is all interesting, but not necessarily to the average controls engineer who actually, perhaps literally, has boots on the ground.
Articles on Connectivity seem to circle around protocols. Perhaps pushing what is currently popular or what might benefit the large corporate interests. The goal potentially being the avoidance of vendor lock-in, competitive cost savings, and interconnection convenience. The industry has searched for that particular Holy Grail for decades.
While it always seems that there are solutions on the horizon, we are concerned with decades of installed legacy equipment that cannot be upgraded. That equipment has worked predictably and is capable of continuing in that way for many more years. Should we force its replacement?
Topics that more appropriately address the daily needs of the engineer are few and far between. Not that there aren’t solutions for that as well. But writing about it is difficult and, frankly, boring. Doing that without an underlying goal of promoting a product is another matter.
With an idea to discuss the pros and cons of Connectivity in some new way, perhaps with a bit more fun, I decided to employ a little narrative/fiction. Here I used a technique that you might recognize from the movie Groundhog Day. Our main character, Sam, has to deal with a real-world controls problem. Perhaps you can identify with one or more of these scenarios? So today, for a little escape, please enjoy the following short story.
Where There is a Will, There is a Way
The café couldn’t have opened in a better location. At least, according to Sam. Since the day it appeared, Sam headed there for his afternoon breaks. The service was fast and he was now recognized as a regular customer.
In fact, it became so routine that others from the office would often come along. Or, perhaps Sam tagged along with the group? He wondered about that now. How did it start?
Today, three of them sat at a table in the sunlight filtered by the vine-covered trellis outside. Cynthia, seemingly lost in thought, looked up and asked, “Sam, so how has your week gone?”
Sam smiled and shook his head. “Oh. Nothing out of the ordinary. I am exhausted. I think I made a friend.”
Cynthia, now curious, jumped on that. “Oh? A friend? Um. Who is she?”
Sam chuckled and grinned. “No. Not that. I wish.”
He continued, “I have been thinking about getting a pet. Maybe a dog?”
Chuck interrupted, “Well, my apartment allows them but charges an extra monthly fee. It would be great to have a friend like that. But then you have to consider all of the things involved.”
While they settled up the tab, Cynthia looked at her watch. “Oh! Got to hurry. I have a meeting when I get back upstairs.”
Chuck eventually, and cheerfully, wished everyone a great weekend as the group entered their office lobby.
Just Another Monday – v1
A noise startled Sam, waking him. He looked at the clock, which had just turned 06:00, and realized that it was time to get up anyway. The noise occurred again. Sam instinctively grabbed his phone. Two text messages awaited him. Both indicated an urgent support request from the Dorsey farm. Sam sighed and said to no one in particular, “And so it starts…”
Upon reaching the office, Sam grabbed the phone and dialed Karl Dorsey. Karl answered and, amongst all of the background noise, he shouted, “Yeah! Hello? What now?”
Sam responded, also loudly, concerned that Karl might not hear him. “Karl? Karl, it is Sam Sturges, do you have a problem up there?”
Karl must have been on a tractor. It sounded like the machine was right there clamoring at the phone. Karl replied, “Yeah, hang on!”
It took a minute before they could continue in a more conversational tone. Karl started back up. “It keeps randomly triggering the pump alarm! I have been up since 2 AM starting the damn pumps manually. Had to do that at least a half dozen times. What is going on?”
Sam put a hand to his forehead. “Karl, can you get to the PC? I can’t get up there until late this afternoon even if I left now.”
About 15 minutes later Karl called back. This time the phone line was quiet. Well, almost. Before Sam could say hello, he could hear Karl yelling apparently at his dog, “Okay! Okay! Quit barking. Here’s a treat. Be quiet.”
Sam finally risked a hello.
Karl jumped right in. “I am going to need that serial cable, aren’t I? I couldn’t find it anywhere. I have got another and I finally connected to the controller. Well… I think.”
The two of them slowly worked through the command line interface. Sam asked Karl to type commands which often needed to be retried, and Karl painfully read back information from the screen that appeared in a language that he could not understand.
Eventually Sam said, “I think it is timing out when reading the level and not getting around to turning the pump on. Let’s try and change this Registry key. Type this…”
Methodically, Karl followed Sam’s instructions. All the while Sam heard Karl telling the dog to get down. Finally, Karl laughed into the phone, “I think he knows it’s you on the phone. He likes you!”
Sam said, “Yeah, tell him I said hi. Okay, this will change the timeout from 60 to 180. Let’s hope that fixes it. Before you hit ENTER, read it back to me.”
Then Sam heard Karl again talking away from the phone, “Yes dear. I know. I know. I got to get this working and then I can do all of that. Yeah, I should have but… Okay. Right away… I know it is important. I’ll get to it before it is too late… But…”
Karl finally confirmed his proper transcription of the command and, after a pause and a big, deep breath, he hit ENTER.
Just Another Monday – v2
Sam was startled out of a dream. He glanced at the clock that had just turned to 06:00. It was just about time to get up anyway. Noticing the illuminated screen on his phone, he reached for it. There were two urgent notifications. Sam complained out loud to no one in particular, “Oh boy! Let’s get right to it. Why don’t we?”
The messages were from Karl Dorsey who owned a farm up North. Sam installed a controller up there almost a year ago. Since Karl’s texts weren’t too cordial, Sam figured he’d get in the car and get started in that direction. He could get breakfast on the way.
While pulling out of the driveway, Sam asked his phone to call Karl Dorsey.
Karl answered and over the engine noise he yelled, “Yeah!! Hello? What now?”
Sam responded in kind, not knowing if he should yell too. “Karl? Karl, it’s Sam Sturges. Do you have a problem up there?”
Karl throttled down the source of the racket. “Yeah, I have a problem. The pump alarm keeps going off. I have been up all night resetting it and manually running the pumps. I am not getting much else done.”
Sam winced and looked at the time. “Karl, I am on my way. It’ll probably be after lunch before I get up there. Look, I am sorry that you are having trouble. See you soon.”
He then left a message for the office: “Cynthia, hi it’s Sam. I have to take the day and head up to Dorsey’s. Can you get me out of the afternoon meeting? Also, check the project budget to see if this trip can be covered under that? I really hope so. We may have to eat it either way. I’ll check in later.”
It was a pleasant drive. The sky was clear and the sun funneled through the trees as the road wound northward. The farm was also in a peaceful and gorgeous setting. That was in stark contrast to the amount of effort that went into it. Sam had seen it in operation. Karl made it look easy. And it was not.
Upon arrival, Sam was greeted by the dog. Karl came out of the barn yelling, “Get down! Get down! Yeah, that’s your buddy Sam. Leave him alone. Go chase the chickens!”
As Sam approached Karl to shake hands and exchange greetings, Karl rolled his eyes and yelled, “Git!” The dog, still elated to see Sam, excitedly leapt in and out of the car but finally stopped barking and ran off.
Karl pointed to the barn. “You know where it is. Get it working. I’ve had to pump manually all morning. Let me know what you find.”
Sam turned to the car to get his laptop, toolbox, and his all-important ballcap. Except, he mustn’t have grabbed his hat on the way out. He was kind of lost without it. But he didn’t have it.
Sometime later, Karl walked into the barn. “So. You can fix it, right?”
Sam, who had just gotten off the phone with the office, was debating how to break the news to Karl that he had to charge him for the service call. “Um. Yeah. I think it was timing out reading the level and never deciding to run the pump. I just have to change the setting from 60 to, maybe, 180.”
Then, thinking that trips up here might be a problem in the future, Sam added, “Let me show you how to do that in case it happens again.”
They finished discussing some details on the latest screen. Sam then typed most of the Registry command to change the timeout, but stopped to cautiously mention the charge for today.
Karl was not looking happy about any of it. Sam reached down to pet the dog, which had been hanging around, and said, “Well, let me see what I can do about that. This probably could be a warranty thing.”
Sam turned back to the screen. “This should do it.”
He then hit ENTER.
Just Another Monday – v3
There was a vibration. It repeated and Sam forced his eyes open. The alarm clock showed 06:00 and silently declared that there was no sense in trying to fall back asleep. It would shortly be time to get up. Sam grabbed his phone, mumbling to no one in particular, “Silent mode, eh?”
Notifications for two text messages from Karl Dorsey adorned his screen. Karl was a customer and had the installation up at the farm almost a year ago. His texts were short and not so sweet.
Sam grabbed something to eat, got dressed and called Karl.
Karl answered. Sam thought at first that he had a terrible connection. Through the noise he heard Karl. “Yeah!!! Hello? What Now?”
Sam wasn’t sure what to do. He tried, “Karl? Karl, it’s Sam Sturges. Do you have a problem up there?”
Almost immediately the racket became a low throb. “Yeah. I have a problem. The damn pump alarm waits until I start doing something else and then goes off. I have been managing the pump by hand all night. Meanwhile, I’ve got stuff to do!”
Sam said, “Well, I am sorry about that. I can look to see what is going on. I am heading into the office. I will call you at 8 o’clock. Make sure the network bridge is turned on. Also, enter the command to enable it for me. The instructions are in the little notepad.”
On the way into the office, Sam bumped into Chuck. Looking a little concerned, Chuck asked, “You are in early. What’s up?”
Sam held the door for Chuck. “Oh, Karl Dorsey has some problem. I need to connect to that controller as soon as I get upstairs.”
Chuck stopped. “Wait! His controller is connected to the Internet? You know, that gets us in trouble. There’s this directive and they’ve started auditing us.”
Sam looked up. “No. Not directly. The customer has to temporarily open the port for me.”
As they reached the escalator, Chuck said, “I had two sites with random failures. We think that was partly triggered by activity on the customer’s networks. They want to disconnect cables now. One location remotely monitors and can’t do that.”
As Sam reached his office door, he said, “Yeah, Karl is a farmer. And if anyone knows about fencing, it’s Karl. He was all about access control. Although, he had more colorful language for that.”
As Sam turned, he almost – uncomfortably – bumped into Cynthia who was holding the phone out to him. “It’s Mr. Dorsey. He says he can’t find the, um… something notepad.”
Sam grabbed the phone. “Hello, Karl. I was just about to call you.”
Just as Karl was about to respond, Sam heard him yelling, “Down! Geez! Go play.” Then back into the phone, “Sam, I think he hears your voice and gets to be a real pain.”
Sam sat down at his computer and grabbed the project folder. “Karl, here are the commands and what to do. Write it down. And here is what you need to undo that after we are done. Don’t forget to do that. That’s very important. You know.”
It took a couple of minutes before Sam could establish an SSH connection to the controller. After several more minutes of typing, Sam said, “It’s got to be a timeout. It goes to read the level and never gets to handling the pump. I am going to change it from 60 to 180, if that is okay with you?”
Karl said, “Yeah. Okay. Sure.”
Karl continued, “Stupid dog. He just took off into the woods. Your controller makes more sense to me sometimes.”
After a lengthy pause, Karl said, “Hey, I might have something for you. When are you coming to install that controller for the silo vent fan?”
Sam stopped to think. “Yeah, that’s Friday, I think.”
He then hit ENTER.
Just Another Monday – v4
Sam felt something wet. Disturbed by that, he awoke from his dream. He looked over at his clock, which had just announced 06:00. He grabbed for his phone. That was the routine. There was a text sent late last night from Cynthia mentioning that he shouldn’t bring a lunch. They were all going to take lunch at the café.
Sam, who was still trying to clear the fog from sleeping, suddenly said, “What are you doing on the bed? What’s with the kisses? Quit licking my hand. Good morning. Let’s go out for a walk.”
Later at lunch, Cynthia, who was again deep in thought, looked up and asked, “Sam, why is your hat so dirty?”
Sam laughed. “Well, Karl Dorsey found it buried by the dog out in his woods, along with some cables and our branded notepad. He gave me the hat back when I went up there Friday for the install.”
Chuck stopped in mid-bite. “Wait? Your hat was buried? At the customer’s place?”
Sam smiled. “Yep. It was. And he gave me the dog too.”
∎
We have all likely been in, or around, situations like this. You know, you program a controller, test it, install it at the site, and set it in motion. There is that little moment of prayer in hoping that it works. Perhaps you keep an eye on it for the day or a week or two. Eventually it is forgotten and added to your growing list of successes.
Then something changes. In our story we are not sure what has changed. Perhaps something seasonal, given that the installation is not more than a year old? Regardless, while the correction itself may be simple, accomplishing it may not be.
The time-proven approach of talking the customer through the fix gets the job done. In fact, this is how INTEG supports the JNIOR. Although in our case, there is likely an intermediary. We work with process control companies, integrators, and distributors. Those fine folks in turn support their customers (and ours).
There is a downside. Not only are diagnostics more difficult, the process also consumes the customer’s valuable time, often disrupting their already busy schedules. That might even make life at home more difficult, as it did for Karl. If you have a definitive solution, however, everyone eventually is happy.
Why is it that the simpler the solution, the further away the controller has to be? Perhaps it is some law of physics or some corollary to Murphy’s Law? INTEG supports customers literally on the other side of the planet. We could be looking at 12,500 miles to the project site.
That travel is an unscheduled interruption. It impacts everything else. There is a hidden cost in that. Perhaps importantly, the wear and tear on the engineer. Urgent visits are expensive, and that premium cost has to be covered by someone.
It is not surprising that we cry out for connectivity and the ability to remotely administer the controller. While that seems like the perfect solution, there is no nirvana. In the story, Sam can easily diagnose and fix the problem. All without interrupting the daily trip to the café.
These days we know that there are considerable issues with that connection. Security issues are the obvious ones. But even rock-solid authentication doesn’t eliminate the computational cost of unsolicited connection attempts. That unwanted network traffic can dramatically affect deterministic performance.
We have two JNIORs with public IP addresses and with all externally facing services exposed. These are there for us to harden the operating system. After years of working with that, there is still an issue involving the computational expense of the cryptography. That can impact performance. The SSH connection must be negotiated before credentials can be submitted. Sufficiently aggressive bots, repeatedly attempting to log in, consume substantial computational resources.
And in our story, Chuck alludes to the CISA Alert that recommends that controllers be, or remain, disconnected from the Internet. That is directly in conflict with the industry drive to connect and publish data for cloud analytics and AI processing.
Of course, remote administration can be, and routinely is, successfully done. Point-to-point connections through Virtual Private Networks, perhaps managed by a security-conscious Network Operations Center, work well. But is that always possible? Would Karl be happy with the ongoing cost? I wouldn’t want to ask.
Clearly, the best way to support a site depends heavily on each situation. A farm is quite different than a corporate campus; a site with a single controller is different from a location having deployed hundreds; and a customer with on-site engineers is different from one with a grumpy farmer.
We learn from the story that, no matter how it is achieved, the successful support of the customer can be rewarding. That must be the ultimate goal.
Where there is a will, there is a way.