Speakers

Fred Zaboli

Principal Engineer

Wunderlich-Malec Engineering

Nicole Williams

Sr. Manager, Maintenance and Engineering

Stone Brewing

In this session, Stone Brewing and Wunderlich-Malec Engineering will showcase the first successful implementation of Sepasoft’s Batch Procedure Module. Going into the project, Stone Brewing hoped to upgrade to a flexible and modern batch system that could handle complex recipes. With the support of Wunderlich-Malec, Stone Brewing easily configured the module to replicate existing processes. Attend this session to learn about Stone Brewing’s quick adoption of Batch Procedure and more project highlights.

Transcript:

00:15 
Mark French: And it's my pleasure to introduce our speakers today, experienced engineers that get things done, and I'm looking forward to their story about their projects. So we start with Fred, who is the Principal Engineer...

00:35 
Mark French: So we start with Fred, who is the Principal Engineer at Wunderlich-Malec. And he has over 18 years of experience in control systems, SCADA, HMI, PLCs, a wide range of experience, and the full breadth of project experience from estimation and implementation, programming. So he's the full package here. We also have Nicole Williams, who's the Senior Director of Brewing Operations at Stone Brewing. So starting out as a brewery process engineer, she's climbed the ranks and fielded several successful projects, leading in those projects technically and in project management. So, we've got a very experienced pair telling us about their projects, so please join me in welcoming them to the stage.

01:27
Nicole Williams: Well, first off, I should probably get it out of the way. I did not bring samples.

01:34
Nicole Williams: But I am here to talk about our new brewhouse control system. So for those of you who don't know, hopefully, you all do, but Stone Brewing is one of the largest craft breweries in the US. We were founded in 1996 in San Diego and are really famous for pioneering the West Coast IPA style of beer that's popular across the US now. We have two main production facilities. Our Escondido, California brewery, and our Richmond, Virginia brewery. Escondido was our flagship brewery. That's the brewery that we're talking about today. Our beers are distributed in 50 states and 40 countries, and we've grown a whole lot since 1996, but we're still an IPA powerhouse. So that's Stone. What we're talking about specifically today is our brewhouse. In the brewing process, the brewhouse portion is where we make wort. So wort is basically unfermented, sugary, sweet water that is later fermented to become beer. We have two brewhouses that both make our wort for us, and there are about 120 barrels each. And one went in in 2005, one went in in 2011 when we expanded the brewery. So definitely starting to get into the aging systems realm.

03:06
Nicole Williams: Okay, so why are we upgrading our brewhouse systems? Well, the first one is obsolescence, the start of all good new control projects. We were buying spares on eBay, so it was time. We looked at our system holistically and really felt that there was no way to replace one component. We couldn't just upgrade the PLCs, or the SCADA system, or the servers. It all had to get to a truly modern system. We needed to go from the ground up to avoid going through the same eBay hunts two years later. So at this point, we got into the realization that to do this, we would be reprogramming PLCs anyway. So there was no real reason to constrict ourselves to a direct migration of our existing SCADA systems and PLCs. So this opened this up to some more possibilities, and we started with, "Well, what's wrong with our new system that we need to fix?" So we had a couple of pain points. One of them was that our system is very canned. This is very common in brewing systems.

04:13 
Nicole Williams: You get a system that's designed for brewing. It does one thing. It's very constrictive. That's kind of what we had, and we found that there was a little bit of trouble getting support for this system because it was so specialty. Also, it was difficult to troubleshoot. The SCADA system itself had very unintuitive configuration. Our documentation from the original system configuration was very limited and not really the SCADA system's fault, but it was all commented in German, which made it really tricky for me personally. Fred speaks German. I don't. All of these things sort of led to some poor integrations over the years as we added in more systems.

05:00 
Nicole Williams: The SCADA systems were a copy and paste of each other. So when we put in the 2011 one, we copied an instance of the 2005 one, and as a result, they had a lot of shared tag meanings and don't talk nicely to each other, and that became really problematic when we had shared resources between the two. And part of this is also just that a lot of the equipment vendors that we work with were overly familiar with how to program on the SCADA system. So every new equipment deployment we did wound up with two integrators. And again, a lot of issues with shared resources. Also, a pain point for us on this was our limited clients. We basically had two clients that we could use for brewing, recipe development, and engineering. This led to a definite lack of progress and innovation on the brewhouse. And all of the above ultimately limited our ability to improve and expand the system. Stone is a little bit unconventional, and one of our core values as a company is giving the middle finger to the status quo. We couldn't do that when we couldn't work on our brewhouse.

06:15
Nicole Williams: Okay. So why the Ignition/Sepasoft solution? One of the big things that we're going for with a new system is we wanted something with lower training costs. We love Ignition because it's so easy to pick up for new engineers. There's so many online resources available for training. And this ultimately lowers our external support costs because we don't need an integrator for every little improvement. We also love that it has unlimited clients and truly drives our ability to improve, gives the supervisors and the brewers more time to work on recipes. Gives engineers more time to keep making the improvements requested of us. We work totally on the team. Ignition can do anything. I love the customization and flexibility that this solution provided for us. Like I said, we were anti-canned software. We didn't want any black boxes.

07:15
Nicole Williams: We wanted full visibility and control for our engineers. We also wanted to empower our brewers. We wanted them to have a lot more control over recipes and over actual control of the process than most of these canned systems wanted to give them. And we wanted to be able to implement all their great ideas. So just the flexibility that the system has provided is really gonna allow us to do that. We also love the improved maintenance troubleshooting of our new system. We've been able to visualize a lot more configuration, interlocking, and alarm data with the new system. And finally, batch recording and tracking was a big deal to us, and being able to customize that and pull out the data that was important to us, not important to whoever built the brewing SCADA system that they were just handing out to everybody. And with that, I'll turn it over to Fred and you won't have to watch me fumble with the remote anymore.

08:18
Fred Zaboli: Thank you, everybody. I'm Fred Zaboli with Wunderlich-Malec Engineering. We are an employee-owned company with over 500 employee. And we are specialized in electrical and control systems and integration services. And we do projects in most vertical industry like pharmaceutical, food and beverage, energy, water/wastewater, oil and gas, and many others. So we got invited to this project, we got invited from Stone Brewing to take a look at their packaging system. It was a job walk, so that's where we got introduced to Nicole. And she start talking about the brewhouse that they have. They have two brewhouses. And she said, "Okay, let's take a look at the brewhouses after this job walk." And we take a look at it, and it was really a closed system. Not much of a documentation, not much of information existed. And she started saying that we have this problem. We cannot change it.

09:35
Fred Zaboli: If we wanted to change it, it has to be a control engineer. So that's where we started thinking about it, that maybe we can have a way of operating that to a newer system, PLC, and also the HMI system and batch system that we have in our existing system. So we started to take a look at this project, starting from designing of this project. So we started to come up with the design of the PLC to rewrite the whole code for one of the brewhouse, replacing the existing hardware to a new PLC with the new I/Os. And also restarted into their existing HMI and replaced it with Ignition. Because Stone Brewing had already had Ignition in the packaging line, and they were happy with that. They wanted to stay with Ignition. And also we figured that there is a really complex batch system existing, and we have to replace that if we wanted to go with Ignition and with the new control system.

10:46
Fred Zaboli: And that's where we proposed having Ignition Perspective for the HMI. That's the first risk we took. And the second risk was to go with this new batch module and Batch Procedure Module. And I wanted to thank Ignition and Inductive Automation and Sepasoft on supporting that they gave us throughout the design of this because we had to run this with Inductive Automation and also we had to run it with Sepasoft to see if it's possible to do this complex system with this new module. And that's where we started to do the actual innovation. So I wanted to point out the implementation of this project that what it takes to do the implementation. So the first phase was to come up with some sort of interface between PLC we have and this new batch module.

11:50 
Fred Zaboli: And that's where Sepasoft implemented this PLC logic interface that they call it PLI. And that was the first one. And it's based on ISA-88, all the states and the commands that it’s sending from the batch module. It's all standard, and this is what most of the PLC brands, they understand that. The second phase was interacting with Ignition and coming up with the screens. So this module provides these UDTs automatically. So as soon as we start developing the units and phases and adding our parameters to the phase, all these UDTs automatically shows up on your tag definition. And that helped a lot. And all we had to do is to give the same naming convention that we have on our batch engine and our PLC so those guys can take a right path and talk to each other. That was the second phase. And then after that.

12:57
Fred Zaboli: That was one of the requirement, going with ISA-88 standard and everything. And that was one of the weak point of the existing system because it wasn't aligned with this standard. And we had units, phases, and steps based on that ISA-88, and if I wanted to give you a size of this system, for one of the brewhouses, we have 10 units. We have 193 phases, and then we have 20 user parameter or settings per steps or per phases. So that makes it a little bit complex system here. And the recipes are kind of big and too granular. Then, after that, if we wanted to talk about the, how we implement this, you, with Sepasoft module, they provide a lot of components to edit your recipes, configure your batch module, and also execute and monitor.

14:03
Fred Zaboli: But and also, you could go to these components and start creating new units, creating your parameters, but for this size of project with 10 units and 193 phases, it was a little bit hard to do it manually. And that's where we got introduced to this new library of script that this Sepasoft module, the Batch Procedure Module provides. So as soon as you install this module, all of these library of scripts are gonna show up, and you can use it either for executing a system, monitoring system, query information from your batch, and also you can use it for configuration, creating units, adding new parameters. And that's what we use. So we came up with the Excel sheet in CSV format, and we use that and in conjunction with the new scripts that we're getting, and it automatically generates all these units and phases of parameters without any error.

15:11
Fred Zaboli: The other thing that I wanted to mention in implementation side is all these built-in components that it might take a long time to get these information from your batch system, but these built-in components like batch monitor, like batch control, recipe editors, all this stuff is built in, and you don't need to write any code or scripting to get to that point. That helps us a lot and it saves us a lot of time. And at the end, I wanted to, in the implementation side, I wanted to mention the custom component that we had to create. So one of the challenge we had on this project was that we had a working system. We all know that it's not fun to touch something that is already working, and everybody's saying that if something works, don't touch it, right? But, in our case, we had to change it. We had to go into it. And one of the things that we had on this project was that we had a lot of feature on the existing system that the operators were really comfortable using that. And we had, it wasn't the best tool to use to do the operation, but we had to give it to the operators so they can continue their operation. And that's where we started to do custom components and Sepasoft Batch and Procedure Module, help us with exposing everything to the Ignition frontend with tags and information that is all exposed.

16:49
Fred Zaboli: And you can use that, and you can show it differently from the components that we already have built into the system. And that was one of the thing we did. So I wanted to show you some of the differences here. As far as the HMI, I said about the Perspective. First of all, Perspective was the good choice for this project. The reason was the quality of the graphic was substantially different in the new system. And also, we follow the ISA-101. As you see in the right side, you have the old system with a lot of color and 3D piping, and everything and in the left side, we have a new system, screens that they are all based on grayscale, high-performance HMI.

17:42
Fred Zaboli: And I'm glad that Nicole and the Stone team, they were on the same page with us as far as the standard of the ISA-101 and high-performance HMI, and we didn't have any argument on that part. And next slide I have here, you can see this is one of the component that they had in their existing system in the right side. They used to call it process cell overview, that it gives you an overview of the whole system that which unit is working, and what unit is doing what step. And it was limited configuration. It was one, some sort of configuration. It was some sort of a third-party software that they were using to monitor the batch. And it was separated piece of software. It was limited configuration. In the left side, you see the replicate of that.

18:40 
Fred Zaboli: And the user Perspective regular table. It's not anything crazy. It's a little bit of scripting in the background to extract those exposed information. But it's a regular Perspective table on the left side. And with that, you get a live process set point change that was one of the point that operators they need to have to be able to change the set point live when one batch is running. So, for example, when they are in a specific unit and running a phase, there's a bunch of set point on that. So they already have those set point on that specific recipe, but when time comes, maybe operators decide to change the temperature higher or lower, so they can do it live from this screen. So that was one of the custom-build component we did. And you can get a bigger view of that here. In the top, you can see that each line has a unit. We have... And then we get the states in the second column, and then the mold, the batch ID. And one of the challenge we had on this project was that Stone Brewers. They run three batch back to back at the same time. So they run one batch from the beginning. When it gets to the end of the line, they run another one back to back.

20:06
Fred Zaboli: So that's how they get the best performance. So we had to show them the whole overview, and they can see which unit is running, which batch ID. And in the bottom, you can see if you click on that unit and that specific phase that is running in the second row on top, you get all of the set points and process value regarding that phase. And so this is some sort of a custom build for Stone Brewing. The next comparison I have here was the way that they had the control recipe to show. So they used to call it a “Control Recipe.” So whenever you run a recipe, you could see where it is but it was very limited as far as the graphic. And you cannot see the whole thing in one page. You have to kind of scroll up and down, left and right, to get to where to see that where is the recipe running, what phase is running. And in the left side, you see the batch monitor screen that we make, that we monitor the running batch.

21:16
Fred Zaboli: And because it's Perspective, because it's HTML based and everything is built into the HMI, they can zoom in, zoom out, click wherever they want, and they can see the whole picture of that batch. And also, they can add user-defined steps, and they can put custom messages to this that we can get to it later on. This is the look of one of the recipes that they want. This is a single one, two-unit CIP unit. We doing CIP here. And those three batches I was talking about that they running back to back, they might be able to run some CIP in between each batch. So it's pretty complex. I wanted to point out the items that made this project a success. First one, I would say, was the equipment icon. So for each equipment, valve, motors, instrument, we have an icon that gives you all the devices' status and everything, and it's available everywhere in this throughout the HMI, Perspective HMIs.

22:31
Fred Zaboli: And then, we have navigation icon in the middle, navigation icons that you can get all the status of the devices under that zone of your navigation. For example, if you wanted to zoom in from your overview screen, you click on this navigation button that it takes you to the Mash Kettle area. So if you have any device under that Mash Kettle area that is in Manual Mode or it has an alarm, you get that on the navigation button so you know how many alarm you have on that area. You have what state of device you have there. In the right side here, you have instrument icon that it gives you plenty of information. You can change the configuration of your instrument. You can change the alarm settings for that instrument. And it doesn't need any control engineers, doesn't need any PLC change. It's all throughout the HMI. And in the left side, you see, bottom left, you see the Unit State Object.

23:38
Fred Zaboli: This is another custom object we created, it's a template in Perspective that you can give the name of the unit of this template, and you can drop it off in any screen, and it gives you the information of that unit that what state is that, when it started, when it ended, and timing wise, how far you are to the end of that phase. And you can drop it off everywhere you want. And we have another component here, the Batch Commands that is available on all Sepasoft Batch Procedure Modules, most of them, that you can command your batch engine and execute your batch engine, you can run your batches, you can abort, reset, stop, and all different sort of commands from everywhere. And this is the main overview screen. So we have Brewhouse One on top, we have Brewhouse Two in the bottom. And this is something new to Stone Brewing compared to the old system. Old system there. It was completely two segregated system, but this one is all in one Ignition gateway.

24:50
Fred Zaboli: And you get the whole idea of what's going on in that brewhouse. And also, because we used the Perspective screens, the graphics were easier to change, and also it was accessible from everywhere, from any devices. And also, we get some site features like Dark Modes and those features that comes with Perspective Module. And also, the good thing about going with Perspective and Batch Procedure Module is that all those components, they are aligned with Perspective standards. So when you change your Dark Mode, it also change Dark Mode on those components. So we don't need to do further changes on your HMI. Here I have a couple of good story, or I should say, new features that we didn't have in the old system. And it kind of came up throughout the project.

25:54
Fred Zaboli: We didn't think about it in the beginning of our project, but it turned out that we needed to have these new features. And one of them is User Messages. And the User Messages, I'm talking about the process messages. So you have alarms, the system is running, you get alarm. So that's a different type of event. But when your batch gets to a specific step and it needs a operator’s attention and action, you need to give operator a message. So we have this phase on Sepasoft Procedure Module that you can set up some messages. These messages are specifically for users, and users that they can change the recipe. They can add as many user messages as they want. And these are all fully customized, and they can put unlimited number of these messages, and it doesn't need any PLC changes or it doesn't need any attention from the control engineers. And it's all in the body of that recipe. And I wanted to get the feedback on this one from Nicole. I know there is good story about that.

27:07
Nicole Williams: Yeah, this was a feature that we were really excited to put into the brewers’ hands. Like Fred mentioned, this is kind of something we had in the old system because we brew a wide variety of recipes. I know we said we're doing IPAs, but we do a little bit of everything. We're constantly innovating. So we have a lot of manual hop additions, manual specialty malt editions, places where the brewers need to intervene in the process. So that's what we were using these before, but they were built in, and some of the messages were weird. They weren't quite in German, but they were a bad translation. And so our version just basically had SOPs that said, when you get this strange message, this is what you're actually supposed to do. Now when they build a recipe, they're building these messages in themselves and putting in useful prompts and useful information. And they've really expanded the number of messages in the system significantly since we added this feature. And Fred, as soon as Fred showed somebody how to do it, I feel like we got 10 new messages into the recipes that day.

28:14
Fred Zaboli: Yeah, they didn't have to. I didn't have to set up any of those messages. And I know, being a control engineers, the way that we write stuff, it doesn't make any sense for operators. And we always have this problem, the description of the alarms doesn't make any sense. It makes sense when you write in the code, but it doesn't make sense when you operate in the system. So for this system, it wasn't easy because I just gave operators these user messages, and they went crazy with it. They add a lot of these messages. They add their own descriptions. And I one day, I was there, and they were in early stage of the project. They're starting to run one of these recipes, and I'm getting lot of messages that I don't understand what it is.

29:02
Fred Zaboli: And I heard that their supervisor adding these messages, and they loved it. So that's something. And because it doesn't have control engineer's attention, they don't have to put work orders. They don't have to take the control engineer’s time for doing this and stuff. So it was a really cool feature that we didn't think about it, but it came up throughout the process. We have one other, two other thing here. Some new features here. We have Value Prompt and Loopback Logic. This is a good one. So it happened that it looks like that the operators, they start, they doing a lot of, they used to do a lot of manual operation, and we knew it from the beginning of the project. We knew that they gonna put a recipe in the Manual Mode, which you can do, and then they used to do manually operating those steps or go back up like 10 steps before and run it from there and do it again.

30:02
Fred Zaboli: And I asked the operator, “Why are you guys doing that?” We, one of the operators, said, “Because you have to take samples on this step, and you have to make sure if you need to do like rinse again, or if you need to boil again or if the boiling is enough or not.” And it needs a lot of manual interaction, and it caused a lot of problem when we were like starting the new system, but they used to do it in the old system. So I reached out to Sepasoft, we talked about this manual operation, and they said, why you guys don't use the Loopback? And they didn't have this feature in the old batch engineer they had. And I brought it up in front of the operators, and I said, “What is Loopback?”

30:52
Fred Zaboli: So he said, “Oh, we can ask you if you wanted to go back to that specific step and do it again and again.” And so this is good. The supervisor said, “This is good because we can, instead of having a lot of SOPs for the operators and they, it lives in their mind, we can just put all this built into the recipe so we can ask operator to go and check and to go and take samples so we can do some action. And we don't have to put the whole recipe in the manual operation.” So that's one of the good thing that happened, and I think it helped the operation a lot.

31:30

Nicole Williams: Oh yeah. I think it's gonna help enormously. And just to clarify, we weren't, those brewers weren't going rogue by putting it into manual that was in our SOP because we didn't have the ability to ask them if they wanted to repeat a step. There was no other way to accomplish what needed to be accomplished in the brewery process with the system that we had. And so now, I feel like we all spend half our lives trying to make sure people aren't putting systems into Manual to work around issues. Right? And so imagine if your baseline procedure is put it in Manual, there's, at that point, there's no fear of putting it in Manual 'cause that's a normal part of the process. So I think this is really gonna help us get away from a lot of manual operation generally.

32:12 
Fred Zaboli: We have one feature here. Recipe Creation. So this is a good one. So we, Nicole, mentioned that we didn't have a way of modifying the existing recipe, or we did have on the existing system, but it was limited, and only a few people could do that. And it definitely needs some changes and modification to the PLC as well because it was really tied to each other. But the new system, the way that this recipe editor component comes, you can do import-export. You can duplicate the recipe and quickly get the new recipe and just change the some sort of some of their settings and come up with a new brand. And to me, it was impressive because this system... I didn't create any of these recipes. I just create one base recipe.

33:14 
Fred Zaboli: And then it didn't take that much time. I think I remember it was in the beginning of the project, we just ran the new gateway, and I was working in my laptop, and I turned my head around, and I saw one of the operators he started to doing like create six, seven recipes and brands right away in half an hour without training. So that told me that this system, it's easier for them to create and edit recipes. And I remember Nicole had to ask them don't do it because we don't know if the main recipe is good enough, but they, they were like faster than us, right?

33:54
Nicole Williams: Yeah. Yeah. It was awesome.

33:56
Fred Zaboli: Yeah. So, that's all of our slide here. And I wanted to thank Sepasoft and Inductive Automation helping us here on throughout the whole project. Designing of that. They help us a lot. They support us a lot with these features that we were asking. I know sometimes we will ask a lot, sometimes, some stuff were short, but we come together, and we did this, and it was really great.

34:38
Nicole Williams: Yeah.

34:40 
Nicole Williams: I don't know how they convinced me that I should be the first one to try the new system. But I will say, no regrets. It's been really smooth. We have amazing partners in Wunderlich-Malec, Sepasoft, and Inductive. So would recommend to a friend.

35:01
Mark French: Thank you, Nicole. Thank you, Fred. Yeah, you were already one of our favorite brewers using our OEE product. And yeah, we only have more reason to drink Stone. So, to kick off the Q&A time, we're gonna open it up to all of you. So if you have a question for them about their project or about using Ignition and Batch Procedure Module to solve these problems, now's the time to kick things off while you think of your questions.

35:35 
Mark French: What is your favorite Stone beer?

35:38
Fred Zaboli: Mine is the Delicious IPA.

35:42 
Nicole Williams: He jumped in 'cause it's my favorite too. You wanted to steal my answer.

35:50
Mark French: Okay, you know which beer to try first. Alright. Any questions from the audience? Yes, Doug?

35:56
Audience Member 1: Do you have any mic or…?

36:00
Mark French: I'm gonna repeat the question after you ask it.

36:02
Audience Member 1: Alright. So, how often are you going in and creating new recipes? It sounded like you initially had operators creating recipes. Is this something where like you've got a new set of hops, and you want to try something new, and you're gonna run it through all your equipment, and you go in, and you offer a recipe, or are you taking an existing one and just manually writing out those user notes?

36:30
Mark French: So the question is, can you tell us about the frequency of new recipe authorship?

36:37
Nicole Williams: As an organization, we make new recipes constantly. But I have never authored new recipes. Neither has Fred. That's in our brewers’ hands anytime we're doing new hop editions or what have you. They're very much in control of that. Which is super cool. I'd say if we... The edge case where we would need to like author a new recipe would be if we're doing a totally new product where they couldn't just copy a recipe. So we make seltzer on one of our brewhouses and not the other. If we wanted to bring seltzer, and that would be a pretty different process. We'd probably step in, author the original recipe, and then let them copy from there.

37:17
Mark French: Great. Yes sir.

37:18
Audience Member 2: Can you talk about the process that you went through to validate the new system that you were getting the same results from the process of the old. How did you fix without minimizing the production downtime? 

37:40 
Mark French: Sorry, Nicole, I gotta repeat it for the folks that are watching.

37:43
Nicole Williams: Oh, sorry. Yep.

37:43
Mark French: So the question is, how did you validate that the new batching system did everything you needed to do like the old one?

37:49 
Nicole Williams: Fred, I will let you take that one 'cause your cutover process was amazing.

37:50
Fred Zaboli: Yeah, so what they did, because, as I said, this system was an existing system, running and nothing was wrong with it operationally. So what we did, we kind of came up with the idea of all the I/Os, and all the instrument and everything was coming to remote I/O racks. So all the I/Os were there. All we need to do was to put the new PLC and show up over the network. And we were taking over all the existing I/O. That's what we did. So we had like 10 BSDs, something around 500 I/Os.

38:29
Fred Zaboli: And in one point of time when we were doing the testing in the beginning, we come in the morning, switching over to the new system, doing our test, switching back to the old system, let operation continue, and the day after, we coming back. So that's how we minimize the impact on production. And I think we didn't impact any production time. We try our best to know that... So we have like a favorable plan always. That's what we did.

39:03
Nicole Williams: Yeah. So because we were able to cut back and forth like that, brewing's a relatively slow process to get to finished beer, we were able to do a single-batch brew. Typically, we'll do three brews into a single fermenter. We've got some smaller fermenters. We do a single-batch brew, let that ferment out, do all of our standard quality checks, sensory checks. And then we knew that beer was ready to go. I think we did a couple single-brew batches like that before we did our final cutover and started brewing full fermenters.

39:30
Fred Zaboli: But as far as testing, we have like a different type of stage of the testing. So in the beginning, we do oil test, and then we get, we did the phase logic test, and then after that, we start doing the water test, like just running water through the system and then get to the actual beer, so we don't waste that much material.

39:52
Mark French: Good call. That would be a crime. Okay.

40:00
Mark French: Yes. Justin.

40:00
Audience Member 3: What does Ignition server architecture look like, and is it redundant, and have you had any problems?

40:04
Mark French: Can you describe your Ignition server architecture for us, please?

40:07
Fred Zaboli: Yes. So the Ignition architecture is they already had a gateway that it was responsible for the packaging line and other part of the brewery.

40:18
Nicole Williams: All of our operations except the brewhouse.

40:20
Fred Zaboli: Except the brewhouse. And what we did, we add a new gateway to the system and we connected the existing historian, so I should say the new brewhouse system is the architecture of the Ignition is front and back architecture. And we're using remote racks. So all the tags, sorry, remote tags that all the tags are posting in the main gateway, because we already have connection to the historian, we already have drivers, and everything ready. And we only add this new gateway with the batch engine on it. And with the remote tags, we were able to get all this information. But as far as redundancy, this setup is not redundant yet as far as this Ignition software. But they are living in a redundant server. So all these are virtual machines. They're historian, the main gateway, and the batch gateway, and they are sitting in a fault-tolerant server that you don't see the difference if it goes down.

41:33
Mark French: Yes, sir.

41:35
Audience Member 4: What were the challenges you faced and how did you overcome those?

41:40
Mark French: Question is what are the major challenges and how did you overcome them?

41:43
Fred Zaboli: Do you want me to start that?

41:45
Nicole Williams: Of course.

41:46
Fred Zaboli: Yeah. Challenges, the first challenge was to study the existing system because it wasn't written, like, it wasn't written as a design document. So we had to study the existing system. We had to find out how many units we have, how they are interacting with each other and then come up with the number of these set points. That old system wasn't easy to export your information. So we had to go one by one and study each phases and see how it works. Because we had an idea where our company, we're doing breweries, we work on other projects, but we had an idea how it works, but you have to replicate what the existing have. So we, the biggest challenge was that to kind of study the existing system and come up with the replica of that. And how we overcome that. We use a lot of simulation. So I have this virtual machine in my laptop that it was set up with Ignition and Sepasoft modules and a PLC simulator right there. And all of the phase logics we wrote, we run it in the simulator before we go to the production. That way, we eliminate a lot of problems when we do the testing. And we didn't want to stop the production.

43:13 
Nicole Williams: Say a few things made it past the simulator, but not too many.

43:19
Fred Zaboli: Yeah, not too many.

43:20
Mark French: Excellent. I passed you earlier. Yes, sir.

43:27 
Audience Member 5: Yeah. I was gonna ask about material consumption. If you've only got 10 units, you've got 193 phases. It doesn't seem like that many units for the number of phases, so normally you have storage units and then you can tune out of that, into your… Then you can usually trace the genealogy through it. So what have you done there in material consumptions and genealogies?

43:53
Mark French: Can you describe the material consumption and genealogy tracking in this project?

44:01
Nicole Williams: I'm not a hundred percent sure I understand the question, but essentially as one batch moves through the brewhouse and each batch gets its own unique identifier, each batch I come to, one batch goes into, or three batches go into a fermenter. So each one has kind of like a modifier on that initial batch coding. But really, the each unit in the brewing process is so linked. We don't delineate. There's no break in the process. It's a pretty straight transfer between one vessel and the next. So that's kind of how our batching works and tracking works.

44:46
Audience Member 5: So more from where it comes out, say silo, which side … how much of that … batch and so the engineer knows what’s actually made it out … with that lot, while tracking through. 

45:00
Nicole Williams: Yeah, we don't have like track or trace or any formal system set up, but all of the lot tracking and the raw materials tracking is logged just through some scripting into a SQL database.

45:13
Audience Member 5: Yeah, I could've done those.

45:17
Mark French: I know some people that can help you with that now.

45:21 
Mark French: Say the word. Okay. Yes.

45:24 
Audience Member 6: Yeah. You mentioned earlier you used a CSV file to generate your equipment model from a spreadsheet. I was just curious, is that like a one-time deal where you started off there and then you changed … or were able to restart that whole configuration from scratch by modifying the CSV? Sorry.

45:49 
Mark French: Sorry, can you describe the… Was it, it was the recipe generation process?

45:54
Fred Zaboli: Yeah, the CSV...

45:54
Mark French: From a CSV file. Can you describe that again?

45:56
Fred Zaboli: Yeah, so the CSV generation, yeah, for the beginning, I just came up with that CSV file and that script to create that for just the beginning. But then, when we got to the Brewhouse Two, I had to reuse that. So I kind of put more comments into it. And I also incorporate that as a button into the screen. So if in the future somebody wanted to create a new phase, not using the components, they can just give the name and the list of parameters, and it's gonna generate that. So to answer your question, no, it wasn't just for the beginning.

46:34
Audience Member 6: Okay.

46:34
Fred Zaboli: It end, we end up having it in the screen. Not for everybody, not for operators, just for the control engineers.

46:43 
Mark French: Right, I think we're at time.

46:47
Fred Zaboli: Yeah. 47.

46:48
Mark French: Okay. We're just a little bit over time. I think we could ask if you guys can hang around for a few minutes to catch any other questions. I saw some hands going up, but thank you, everybody, for coming. And thank you, Nicole and Fred, for sharing this successful project