Introduction

The Wastewater Treatment Division of the Metropolitan Sewer District of Greater Cincinnati recently established a two-tiered predictive maintenance (PdM) program. The two-tiered PdM program has both centralized and decentralized components to serve its seven (7) widely separated plants.

The PdM program plan was developed with internal staff and facilitated by an external subject matter expert through a series of workshops, plant site visits and personnel interviews. The centralized component consists of a team of predictive maintenance technicians (PdM team), trained and certified (where certification requirements are established) in at least one predictive technology, initially. Team members are encouraged to be trained and certified in additional technologies as their workload permits, in order to provide backup for each other during absence from the team for vacation, training, illness or other need. The PdM team members are supported to become as competent as possible in assigned technologies. The PdM team is centrally based at the largest of the plants and members are co-located in the same office. Team members have dedicated vehicles for transportation to other plants. They travel whenever the need arises for routine periodic data collection, post-repair or new asset baseline data collection or support of maintenance crew members in diagnosis of equipment for more detailed identification of faults than is possible with tools routinely available to crews locally.

The decentralized component of the PdM program involves local plant maintenance personnel who receive training and support from members of the PdM team. Local maintenance crew members are trained to use easy-to-learn and easy-to-apply predictive maintenance instruments. Typically, local crew personnel can be trained to use them in less than a full workday. These instruments may also be used by the local crew after problem correction and reassembly to conduct post maintenance testing to confirm that conditions and performance have been returned to normal.

The presentation will describe:

• Technologies selected and employed at each level of the two-tiered program
• Benefits of a two-tiered PdM Program
• The processes by which the program elements operate
• Means of communication of PdM program information between participants and all potential users
• Reasons for and problems encountered while setting up an in-house PdM program (as viewed by the PdM team leader)
• Possible additional pitfalls to be avoided when trying to establish a two-tiered (or any type of) PdM program or bring a contractor provided program in-house
• Elements of a plan that can be used to start or convert to a two-tiered (or any type of) PdM Program.

Background

The decision to establish as internal PdM program at MSD is aligned with the Strategic Maintenance and Reliability Plan that supports the WWT mission and vision. This plan has six (6) goals and objectives stated below:

Mission – Preserve process functions so we protect the health of the citizens and the quality of our environment.

Vision – Maximize plant throughput – treatment, at the most effective cost.

1. Focus on core business. Prevent and correct failures. Reduce special projects.
2. Training – Invest in maintenance and reliability program and specialized training.
3. Communication – Increase involvement and awareness through leadership listening.
4. Reliability – Generate proactive work through improved maintenance strategies and a living program to improve uptime.
5. Maintainability – Reduce downtime through centralized and decentralized predictive maintenance program.
6. Plan and Schedule – Planner/Schedulers are focused on the future and capturing work history.

Fulfillment of goal and objective number (#) 5, above, is the primary focus herein. Goal and objective #1 above will be positively impacted when the full force of the predictive maintenance program described below comes into effect. Training for maintenance crews and PdM team members in various PdM technologies supports goal and objective #2 above. 

Additional reasons for deciding to establish an internal, two-tiered PdM capability were to integrate PdM technologies into the culture, reduce cost, develop core competencies and to reduce the time between completion of a repair of an asset subject to monitoring with one or more technologies and the confirmation with post maintenance testing that the repair was successful. This, in turn, hastens the time when an asset can be restored to service. It is most important to the overall mission of MSD to have available (to the extent of its designed capacity) all of the equipment needed to process wastewater in order to protect the health of the citizens and the quality of our environment.

PdM Technologies employed

The PdM program centralized, first tier employs a full-time PdM team, initially consisting of a team leader and four (4) team members with mechanical and electrical skills and maintenance experience. Between them they collect, analyze and/or make other use of data from the following technologies:

• Vibration Analysis
• Infra Red Thermography
• Ultrasonic Analysis
• On-line and Off-line Motor Testing
• Lubricant and Wear Particle Analysis (separately organized and operated as described later in this paper, but with results visible and available for use by PdM team members)

Assignment of technologies to individual team members (including the team leader) is based upon prior experience or previously held PdM technology certification(s). It is expected that team members will become certified in more than one technology, eventually. This will be a progressive process, starting with one technology each and after reaching a certain level of competency in it, beginning to learn to apply and eventually be certified in a second, third or even a fourth. The development of skills in application of the technologies listed above depends upon not only initial training, but also many months of application on-the-job. Where certification levels mentioned above are defined (by such professional organizations as the American Society for Non-destruction Testing – ASNT), the normal time expected for a person to achieve basic certification (Level 1) is about one (1) year. Another year is typically needed for a Level 1 certified person to achieve Level 2 certification and yet another year is required to achieve Level 3. However, learning about how to apply multiple technologies can occur simultaneously. For example, after achieving Level 1 certification in Vibration Analysis, and while pursuing Level 2, an individual can begin Level 1 in a second technology such as Infra Red Thermography. A lot depends upon the number of assets being monitored and defects encountered. With over 15,000 assets distributed over seven wastewater treatment plants in the Greater Cincinnati area, MSD offers an ideal environment for PdM Team members becoming certified at least to Level 2 in multiple technologies in a relatively short time (2 or 3 years). Level 3 certification may or may not be pursued, since the requirements for certification at that level in most technologies where it is needed, tend to stress PdM management rather than technology principles and application.

The importance of multiple technology skill acquisition cannot be overstated. Many defects in equipment and systems provide a variety of indications of their existence, but no one technology can detect or provide complete, precise definition of all of them. The analysis technique called “correlation” uses symptoms indicated by two or more technologies (or within the same technology family) to define and help diagnose more accurately the nature of the problem being detected so that planning for its correction can be most effectively performed. In most cases planning should begin soon after the defect is first detected and confirmed at an incipient stage and (ideally) well before complete failure and loss of asset function.

The team uses state-of-the-art, specialized (i.e., vibration, infra red, ultrasonic, etc) vendor-supplied computer software programs to analyze data collected from assets included in the program and communicates these findings and related information using a comprehensive, multi-technology predictive maintenance management software (PDMMS) program described later in this presentation.

The second, decentralized tier of the PdM program involves local personnel whose primary function is equipment preventive and corrective maintenance. Within MSD, maintenance crews at each plant or group of (smaller) plants are led by crew leaders. Crew leaders report to supervisors of maintenance, who typically have the most years of experience in maintenance and who have been selected from the pool of candidates who have passed civil service exams for supervisory positions.

The technologies employed by local maintenance crew personnel are simpler, less sophisticated predictive technology tools than those of the PdM Team. These technologies include (but, ultimately, may not be limited to):

• Vibration Analysis with “green, yellow, red” severity readouts
• Infra Red Thermography “guns” with integrated visual imaging and digital data transfer capability
• Ultrasonic Testing with Decibel readout and digital data transfer capability
• Laser Alignment Equipment for rotating machines

The three (3) tools listed first above are able to provide in-situ, field indication of degradation, but have limited diagnostic capabilities. Only alignment tools require in-depth knowledge of how to do more sophisticated analysis so that corrective measures can be taken immediately. The purpose of all instruments provided is to empower decentralized maintenance crews to determine, with more data than their five senses, when equipment condition or performance is normal or abnormal and in some cases what defects are developing. After completion of repairs, the same instruments should be used to confirm that asset condition and/or performance have been returned to normal status so it can be turned over promptly to Operations for use.

Local maintenance crew personnel also support lubricant and wear particle analysis (L&WPA) by collecting and transmitting liquid lubricant samples and performing lubrication using ultrasonically aided grease guns as has been done in the past. Laboratory reports of results from analysis of liquid lubricant samples are directed to each WWT plant for appropriate follow-up action. They are also provided to the PdM team for use in correlation analysis (with data from other technologies), pattern recognition and relative comparison analysis across common assets with the same operating profiles in multiple plants.

The L&WPA program at MSD was established over a year earlier than the two-tier PdM program and has already gained recognition for its success. Rather than integrate a well-functioning L&WPA program at the development stage of the PdM program, it was decided to keep them separate for the near term. Only the results of analysis from the outside laboratory to which samples are sent are integrated and made readily available so PdM team members can use them in correlation with results from other technologies.

The addition of laser alignment tools at the decentralized local maintenance crew level of the WWT organization is consistent with research findings from the Maintenance and Reliability Center at the University of Tennessee that shows that shaft coupling misalignment, even when well within manufacturer allowable specifications, is a major contributor to bearing life reduction and premature failure. 1 Laser alignment has been shown to be more easily performed than earlier mechanical methods (such as the reverse dial indicator method). This leads to more precise outcomes with fewer labor hours. Objectives and goals for laser alignment technology are delineated below.

• Maintenance Crews: - To provide the most productive and accurate available tools for equipment alignment so as to extend bearing and coupling life and overall asset reliability.
• PdM Team: - No direct goals - since this alignment capability is provided at the maintenance crew level of the MSD organization. Indirectly, however, rigorous application of laser alignment will have an impact on equipment reliability by reducing the number of bearing and coupling failure “finds” reported by the PdM Team and reducing the overall vibration levels for MSD assets.

The Predictive Maintenance Process

The process of predictive maintenance being conducted by MSD for WWT plants is depicted in the figure below.

As you view the MSD WWT PdM Process, keep in mind the following:

MSD established three categories for work orders WO’s in order to better track key performance indicators (KPI’s) for:

• Corrective Maintenance (CM WO’s – unexpected repair or unexpected like-kind replacement
• Preventive Maintenance Repairs (PMR WO’s) for scheduled preventive maintenance inspections (non-PdM tests) and related follow-up work to remedy any defects found 
• Predictive Maintenance Repairs (PDMR WO’s) for follow-up work to restore degraded (but not usually fully failed) conditions found using PdM tools in the hands of either the PdM team or local maintenance crew PdM practitioners.

• PDMR’s appear in both the Computerized Maintenance Management System (CMMS) and the Predictive Maintenance Management System (PDMMS). The link is the work order (WO) number which appears in both. Asset numbers, and terminology are identical in both systems.
• MSD’s WO priority system has five (5) levels – Priorities 1 thru 3 are “routine,” indicating action can/should be taken within the planning horizon starting in the next week or more from date of origination. Priority 4 requires action within the current week and 5 (emergency) requires action immediately – either category breaking into and pushing aside already scheduled work of lower priority, if necessary. A well-functioning PdM program will yield very few high priority WO’s. This is because, when degradation is detected at an early stage in its inevitable progression towards complete failure, orderly planning and scheduling (e.g., when maintenance crew assets are available) and other factors (such as weather forecasts in the event of wastewater treatment facilities) are favorable so that repairs can be accomplished when MSD personnel, - not the asset – dictate its needed.

Processes also have been developed for periodic routine PdM data collection, post maintenance testing, baseline testing and cost avoidance determination.

PdM data collection will be discussed in the latter part of this presentation in a segment prepared by the PdM team leader. Post maintenance testing is discussed below within the context of a benefit of a two-tier PdM program.

As described above, baseline testing by the PdM team has been separated from post maintenance testing. Post Maintenance Testing (PMT) is now done at MSD by the maintenance crews. PdM baseline data are collected by the PdM team after a successful repair near the beginning of a new operating cycle, so that trend analysis and other aspects of the more sophisticated PdM software tools can be re-started. That minimizes delays in returning assets to service.

An important feature of the most successful predictive maintenance programs in any industry, government agency or utility is the continuous accumulation of cost avoidance data on equipment monitored. Cost avoidance is calculated by subtracting the actual cost of any repairs performed before complete failure occurs, because of knowledge applied through use of predictive technologies from historical cost of repair determined when previously made after complete failure of an asset. The accumulated values are compared annually to total cost of the PdM Program (including labor and benefits cost of PdM team, personnel training and certification, contract costs such as for Lubricant and Wear Particle Laboratory services, costs of computer services and/or software (including upgrades), calibration and repair costs of the program equipment, consumables, etc., and capital acquisition cost associated with PdM technology purchases, prorated over the equipment expected life cycles). Metric integrity is assured because the run-to-failure estimated repair costs and the actual cost data are provided by maintenance personnel. The PdM Team merely takes the data given and calculates the difference.

Note that MSD is in the early stages of this process and isn’t attempting to collect cost avoidance data on all PDMR’s completed. It is determined just for those meeting the threshold value for higher cost items that, if left unattended, would have run to complete failure. This is done intentionally to minimize the efforts of maintenance crew personnel and others who are asked to make these estimates based on their past experience or by researching actual costs from historical data. It follows advice given by hosts during a benchmarking visit conduced early in the development of the program described in this presentation.2 Cost avoidance using only the high-cost items has been found to be more than enough to justify a PdM program in terms of its return-on-investment. Pursuing low cost event financial data isn’t worth the time and aggravation of those who would be requested to provide it.

The PdM Cost Avoidance Determination Process is depicted in the diagram below.

Benefits of a Two-Tiered PdM Program - There are substantial benefits to a two-tiered PdM program at both levels. These benefits are over and above those for a “traditional” team-only or decentralized, nonintegrated approaches. Under this two-tiered arrangement with multiple technologies, the strengths of a highly skilled PdM team with an array of sophisticated data collection hardware and analysis software suites will be able to support and collaborate with the maintenance crews equipped with simpler, but sufficiently sensitive tools for detecting and confirming indications of problems developing in their plants. In addition, after a repair, maintenance crew PdM practitioners are able to determine with sufficiently sensitive and accurate PdM tools whether or not recently conducted actions taken were successful. Some of these benefits are discussed further, below.

• Local maintenance crew personnel are empowered and equipped to declare an asset ready for return to operational service. A PdM capability provides maintenance personnel with their own quality assurance tools. This is done by having the local maintenance crews responsible for post maintenance testing. Post maintenance testing is performed to determine if a restorative or mitigating task has been successful in fixing any reported problem(s) as well as to ensure that new problems weren’t introduced in the course of maintenance. In the event post maintenance testing reveals a condition considered abnormal (e.g., vibration level not in the “Green” zone or an infra red image shows a hotter-than-normal or colder-than-expected condition), help from the PdM team can be requested to define what may have gone wrong during the maintenance performed so that it can be remedied prior to turnover to Operations. It helps avoid turning equipment over to operations only to find out that the job isn’t over and that more needs to be done.
• There is division of labor and responsibilities in employment of predictive technologies between PdM specialists on the PdM team and PdM practitioners on the maintenance crews. This can pay dividends in at least three (3) ways.
  1. The maintenance crews, equipped with simple, but effective tools can do the post maintenance testing and make the “Go/No-Go” calls with a reasonable level of assurance of being right. This boosts their confidence and self esteem.
  2. Having a two-tiered program helps identify candidates who have the aptitude and interest in becoming a PdM team member should a vacancy become available 
  3. A two tier approach relieves the PdM team of the need to perform this post maintenance testing task under time constraints (and pressures) needed to return the asset to service as soon as possible.

Another advantage of a two-tiered PdM program over other approaches is that it provides added  opportunities for cooperation between PdM team and local maintenance crew members. For example, when baseline testing - by PdM team member(s) - is being conducted, it is highly desirable that the cognizant maintenance crew PdM practitioner(s) be present with their PdM tools to take readings at the same time. This very often allows the maintenance crew member to compare readings taken on the simpler tools to those taken by the PdM team member(s) on their more sophisticated equipment. This will educate both groups on each others’ capabilities (and limitations), and enhance future communications and cooperation.

This is also important when the PdM team brings tools to the plant such as off-line and on-line motor circuit analysis suites for which the local maintenance crew has few or no comparable, simpler versions. The ideal arrangement for off-line motor circuit testing is to have any lock-out/tag-out by maintenance crew (electrical) personnel done just in advance of the PdM team member making up test lead connections and collecting data. This allows for more rapid data acquisition and minimizes time off-line for assets being monitored. It also provides an opportunity for maintenance crew electricians to become knowledgeable about the technology being used, so that when some defect is suspected, the crew can communicate to the team that a particular capability is believed to be needed to diagnose the asset.

Means of Communication of PdM Program Information between Participants – MSD of Greater Cincinnati chose to communicate PdM program information using a securely linked, web browser – based software service.3 There were several reasons for this decision including, but not limited to the following:

• Wide dispersal over the geographic area in and around Cincinnati, Ohio of actual and potential users (maintenance planners, crew leaders, supervisors of maintenance, PdM team members and senior managers of the Waste Water Treatment Division, the current PdM monitoring, training, mentoring and certification contractor for vibration analysis and infra red thermography4 - and in future, hopefully - electric motor and pump repair shops and other inside and outside repair and support organizations such the lubricant and wear particle analysis laboratory).
• Concerns from the Cincinnati Information Technology department to allow access by outsiders through established firewalls to the MSD CMMS or other internally maintained programs (more on this later in this presentation).
• The overall initiative to increase MSD plant availability, reliability, maintainability and performance that includes many other (sometimes conflicting or currently developing) elements, including:
  • Upgrading of the Computerized Maintenance Management Software (CMMS).5 It was determined that while the current and upgraded CMMS could provide some of the features included in the PDMMS, that there was potential for “bogging down” CMMS with PdM data to the detriment of CMMS performance in handling all CM. PMR and PDMR WO’s.
  • Adoption of revised planning and scheduling processes6
  • Continuing Classical Reliability Centered Maintenance and Experience Centered Maintenance Analysis.7

The PDMMS selected has a number of built-in program metrics and status report features that are related  to the work orders (PDMR’s) originated as a result of PdM program “finds.” These include:

• Dashboard level (pie charts showing percentage of critical (Red – high priority), warning (Yellow – medium priority), suspect (Blue – low priority) PDMR’s currently outstanding overall and within each plant)
• Detailed condition reporting features that provide essential information needed by users to understand where and what problems currently exist in all the plants. The detailed condition reports provide for easy insertion of PdM vibration and ultrasonic spectra and related data, infrared and visual images, off-line and on-line motor circuit test results including graphs, histories and other indicators of degradation.
• Route scheduling and compliance features that show what has been completed by specific technologies and plants and what is overdue to the point of specific assets being declared “at risk” for lack of monitoring within scheduled periodicities or actual degraded condition(s) detected.
• Metrics such as number of PDMR’s opened over time and average days to close by month, mean time between “failure” – defined in this case as detected, degraded condition(s) meeting the threshold for PDMR WO initiation – as well as actual cost of repair verses what the estimated cost would have been if the asset been allowed to run to complete failure (discussed earlier in this presentation as the cost avoidance metric) used to determine “return on investment (ROI)” of the PdM program.
• Repair histories (failures and repairs) for individual equipment and equipment groups being monitored for all assets within the scope of the PdM program as well as most frequent, most recurring and most costly modes of “failure” per the above PdM-based definition.
• Tracking of actions required to eliminate or mitigate causes of failure (that are separately determined from Root Cause Analysis and Defect Elimination processes that are operated outside of PDMMS).

An immediate benefit of the availability of the web-based PDMMS was the adoption by the vibration and infra red monitoring (and training and mentoring) services contractor of its use instead of the quarterly reporting vehicle previously employed for MSD. While the previous method was well received by users of report content, the web-based findings were more easily accessible to users granted access to the PDMMS and actually cost less for the monitoring contractor to enter than the reporting vehicle in replaced. This immediately reduced the cost of the on-going monitoring contract. In addition, the data on vibration and infra red (from the contractor) were thereafter located in the same place as findings from ultrasonic analysis and on-line and off-line motor circuit testing (from PdM team and in some cases the maintenance crews and planners). Further to this end, lubricant and wear particle analysis laboratory results, previously transferred into the monitoring contractor’s earlier reports are now entered into the same web-based PDMMS reporting vehicle as is all other PdM information. Currently it is transferred by PdM monitoring contractor personnel. Ultimately, it could be entered directly by lubricant and wear particle analysis laboratory personnel.

Reasons for and Problems Encountered in Switching to an In-House PdM Program (As described by the MSD WWT PdM Team Leader): While most companies are satisfied with the performance of their out-sourced PdM program there are some of us that recognize they have a pool of in-house talent that in time and with training can do it, that budget cuts and pressures have forced limiting, reducing or defunding PdM contracts and/or they just aren’t getting the desired results for the time and dollars expended.

My experience in our move to an in-house program is a mix of the above. Using local maintenance crew personnel, we have been doing ultrasonic analysis and on and off-line motor testing for several years to help complement our contractor managed vibration and infra red (IR) thermography program. The problems at MSD (assigning no fault to our contractor) were with PdM data collection, correlation, communications (particularly feedback and follow-up on PdM initiated work orders) and planning and scheduling.

For point of reference it should be known that our contractor has no access to the in-house technology and CMMS databases. This is because of strict firewall maintenance imposed by information technology (IT) managers who enforce security measures needed in today’s potentially hostile and sometimes dangerous cyberspace environment. This makes correlating data very difficult or impossible for those outside the firewalls. All in-house technology databases were run by our planners who issue work orders based on the red (urgent) or yellow (suspect problem) status described as a result of the in-house testing but without much internal analyzing. The planners also handled writing the work orders from the quarterly PdM report (provided under our earlier program via a file sharing service) from the contractor. Opportunities for data correlation were often overlooked or too difficult to perform by those not trained and dedicated to doing so.

Our former program operated under a pretty basic plan. We had the contractor perform vibration analysis and IR thermography on a quarterly basis on assets deemed “critical” to the waste water treatment process. An asset list was sent out the week before contractor quarterly visits. The week of their visit to a large plant, during the first one or two days, they would collect data on all running equipment. The second or third day it was up to operations to switch “missed” equipment so it will be running the last days of the week of data collection. Smaller plants took less time. While some of you are saying, “That sounds like a decent plan,” there are some out there that can see all the inherent flaws it took us years to try to correct and for which we had to devise “work-a-rounds.” 

Not having in-house PdM technicians skilled in even basic vibration analysis and infra red thermography became problematic almost immediately after the start of the contractor-based PdM program several years ago. If critical equipment was unable to be switched by Operations (for any number of - sometimes mystical - reasons) that piece of equipment was missed until the next quarterly contractor visit (or longer) to that plant. To be fair, when any equipment is first brought on-line in a WWT process stream, it sometime takes days to get the whole systems “settled” and running smoothly again. Or, if a piece of equipment only runs during the wet season, let’s say, and it doesn’t rain that week the contractor is there, we miss an opportunity to collect data. There are many difficult-to-overcome scenarios that I can come up with, but I’m sure you get the idea.

Delays in post maintenance testing (PMT) was another issue we had. When the contractor identified a problem, the in-house maintenance crew would make the repair. Well, how do we know if the reported defect was corrected? We couldn’t let a machine that wasn’t repaired properly run (or remain idle) for full  quarter before PdM readings were taken! We decided we would call in the contractor to do PMT and, if satisfactory conditions were found, to take new baseline readings before turning it over to Operations. It could take a couple days to get the contractor to the asset to be tested, results analyzed and reported. But, in developing that process we discovered that we were calling them in for PMT on only one item several times a month. That started to get pricey, according to management thinking. So, we built another work-around to minimize cost. We would call in the contractor when we had more than one asset ready for PMT. Well, this put us right back in the same sinking boat; we just had to use a bigger bucket to bail.

After several years of trial and error, and countless meetings about “improving” the program, it was decided that we needed an effective in-house PdM capability.

MSD engaged an individual with over 40 years of PdM experience to guide us in this most recent PdM program development.8 After an initial survey of what already existed and what was working and what wasn’t, a five year plan outlining tools, training, software, workflows and budgets was developed. What  came out of that was a two tiered system of low-end and high-end PdM technology tools and users serving all of the seven plants within MSD of Greater Cincinnati.

High level tools (vibration analysis suites, IR thermography tools and analysis software, ultrasonic analysis devices and associated software and on-line and off-line motor circuit analysis suites) were placed in the hands of PdM team members selected from the in-house maintenance crews based on their past experience and aptitudes. Previously, motor circuit testing had been carried out by maintenance crews, but with highly variable results. Again this was because of crew personnel turnover and the need for substantial training and field experience in order to become proficient in use of the equipment and to perform proper interpretation of data from testing.

The PdM contractor supports a training, mentoring and certification program with appropriate milestones, number of hours and courses needed to make the team fully capable in vibration analysis and infra red thermography.

Quarterly contractor visits for monitoring became quarterly “shadowing” sessions where initially the PdM team would observe the data collection process and learn the basics of the technologies in which they were ultimately going to become certified. After one or two quarters, the PdM team started collecting the data - with the mentoring assistance of the contractor in some cases. Getting to this point was a huge milestone in the program. Instead of missing a lot of equipment each quarter the PdM team collects data on the missed equipment throughout the quarter, sends the data to the contractor for analysis and entering findings into the PDMMS. The PdM team initiates work orders in the CMMS for crews to repair and conduct PMT. This more relaxed but complete approach to data collection has been well received by Operations as it is less aggravating then the earlier method.

Vibration meters, simple infra red (IR) “guns,” fairly sophisticated laser alignment tools, simple-to-learn ultrasonic detection devices were acquired after some research by crew leaders. They selected from the many available tools in the marketplace those that are best suited for waste water treatment assets and maintenance crew personnel who would be assigned as PdM practitioners. Ultrasonic sensor-aided grease guns had been acquired previously and had been used by local maintenance crews for some time. This was continued. Sophisticated ultrasonic analysis devices previously issued to the maintenance crews but not used very much because of their complexity and crew member turnover were transferred to the PdM team. An existing, well run and managed outside laboratory supported lubricant and wear particle analysis program remained in the capable hands of local maintenance crews and their planners.

Maintenance crew PdM practitioners who received the simple-to-learn tools (that require only a day or two of training) began using them to conduct PMT after corrective repairs of degraded conditions found by the contractor and the PdM team.

Note: I cannot stress enough how important it is to have people with the desire and drive to make the PdM program work both within the PdM team and the local maintenance crews.

Our average time to complete PDMR work dropped from over 73 days down to 6, a huge improvement. The graph below shows real data indicating progress as of end of 2012 only six months after the PdM team was formed and four months since local maintenance crew initial training began.

Annual alignment PM’s were developed and workflows made to get baseline readings on newly installed equipment and rebuilds or major repairs. Once alignment is completed a task is created for the PdM team to acquire new PdM baseline readings.

After the newly formed team was in place, a predictive maintenance management software (PDMMS) contract was established to allow MSD personnel engaged in the PdM program manage and correlate data from all the PdM technologies resulting from efforts of the team, outside contractor and maintenance crews. All routes in CMMS were inactivated and put into the PDMMS for condition assessment tasks. Asset identification and descriptions used in the CMMS were duplicated exactly in the PDMMS, to minimize confusion of users of both programs. Some minor customization was done by the PDMMS contractor on their software to allow for entry of data needed for cost avoidance metrics.

The PDMMS is web based and operates outside our computer network firewalls. However, because it operates outside of the firewalls, some manual transfer of data (such as work order numbers, etc.,) between CMMS and PDMMS must be done. This requires daily effort on the part of PdM team members. Hopefully this will be remedied (or mitigated) when the upgraded CMMS is fully functional. Other features of our PDMMS are described in more detail earlier in this presentation.

Happily, outside contractor9 personnel supporting the CMMS upgrade previously mentioned were on site when the PDMMS was activated. They helped transfer PdM “route” data with the exact plant asset identification and hierarchy into the PDMMS so that they “mirrored” each other. This eased the administrative burden the team (and particularly the team leader) had to bear in order to get the in-house PdM program up and running. The administrative burden remains high, however, because of the increased number of PdM “finds” which our advisor warned us is normal during the first year or more after startup of a PdM program. There are many other office and team related duties, such as attendance at meetings with various personnel at all levels of MSD, that interfere with team leader training, mentoring and gaining field experience with the technologies assigned.

Classroom training by the monitoring contractor and technology vendor instructors along with In-field training and mentoring is proving to be invaluable. Once hours of on-the-job practice are met, certification in the assigned technologies is required. This is overseen by a Level 3 PdM specialist from the outside contractor providing vibration analysis and infra red thermography services until the PdM team takes over completely. After 2 years all PdM team members are expected to be at least Level 1 certified in at least two technologies. In some cases Level 2 can be achieved in one or more areas.

In my opinion the most important part of a two-tiered PdM system is the involvement of the local maintenance crews. Instead of an “almighty” PdM team coming out and telling them what to fix and whether they did their job correctly, it’s more of a Unified Theory approach. The PdM team says (via a PDMR work order) “Hey, we found this, can you let us know when you corrected it?” The local maintenance crew goes out, troubleshoots the problem, makes the repair and checks it with their low level PdM tools. This gives them experience with the technologies and gets them involved with the PdM program. This, in turn, helps change the maintenance crew from a reactive mindset to a proactive mindset. Instead of hearing a pump making noise and tearing down for rebuild they are more likely to grab their low level tools and try to further define the source of the noise so that a simpler (and less costly and time-consuming) repair can be made instead of a costly rebuild. If they need more sophisticated diagnosis, the mechanism is set up for quickly getting PdM team support. After repairs are complete, the crew can do its own post maintenance testing, because the PdM tools they selected are sensitive enough to give them assurance they are making the right calls. The crew can then immediately notify Operations that the asset is ready for use.

After notification via the CMMS that the asset is ready for operation, the PdM team can schedule baseline data collection individually through Operations or incorporate the data collection in the next scheduled “route” to be run that includes that asset at a particular plant, if that is coming up sooner.

The vast improvement in the overall program can be seen in the increase of PdM work per quarter. In the first quarter alone there was an increase 9% in proactive work, partially because the PdM team is adding assets to the monitoring routes as they see the need under a new definition of what is “critical” to have available within MSD plant to fulfill its mission and what assets should be added to reduce the cost of repairs through early detection of degrading conditions. Thus, more assets are being monitored and more issues being found as a result.

We are less than one year into our five year plan and I can’t wait to see what the future holds.

Possible Additional Pitfalls to Be Avoided When Trying to Establish a Two-tiered (or any type of) PdM Program or Bring a Contractor Provided Program In-house - If your organization has an established contractor provided PdM program that you want to bring in-house, it should be done only after careful analysis of expected benefits and problems that are likely to be encountered. For example, it takes a long time for PdM technicians to become proficient in some technologies (particularly vibration analysis and infra red thermography). Other technologies, such as ultrasonic analysis, may be easier to learn. Actually having a competent PdM contractor already engaged with your assets will make the transition to an in-house program easier and quicker, if the contractor is willing and able to train and mentor your personnel in the technologies already being applied. If the contractor has Level 3 specialists, they can also help with certification in their specialty areas, if this is important to your organization. It should be noted (and many PdM services contractors will attest to this) that most attempts to bring a PdM program in house fail sooner or later, not because of the contractor, but because of internal factors and issues such as personnel turnover and failure to select personnel suited to the practice of predictive maintenance. The factors and issues include but not limited to:

• PdM candidates’ lack of computer literacy,
• Candidate inability to learn complex PdM technologies
• Lack of appreciation by managers, supervisors, team candidates and co-workers of the difficulty of achieving competency in a complex PdM technology
• Failure of managers and co-workers to appreciate that while a fair portion of a PdM technician’s work is done in an air conditioned, comfortable, office-like setting in front of a computer and much of the rest of the time in the field is with fancy electronic packages (rather than wrenches, hammers and screw drivers) that the job is every bit as demanding as those of maintenance crew personnel
• Failure to create and maintain current a PdM program master plan (discussed in the next section of this presentation)
• Failure (of management – same for all other items below)) to establish and defend over the long term an adequate budget for all aspects of a PdM program
• Failure to educate and orient management, supervisors and co-workers on the benefits of a PdM program to them collectively and individually
• Failure to continuously calculate financial justification (return-on-investment) and document other tangible and intangible benefits of a PdM program in order to see its true worth year after year 
• Failure to provide for retention of PdM technicians after they become competent in assigned technologies
• Failure to establish a succession scheme for PdM team personnel who retire or who will move on to jobs having greater responsibility when the incentives or time is right.

The last item above is particularly important. If the ideal candidate(s) are selected for a PdM team, management must expect that sooner or later at least some of them will be able to move to better paying, higher level positions, also. Ideally this will be within the current organization, where they should become “champions of PdM.” In the worst case they leave with only a two week notice and go elsewhere, and it takes months to identify novice candidates or hire partially experienced or even certified replacements and get them up-to-speed. In the meantime, monitoring languishes, reliability may decline and, if its true worth hasn’t been documented, the PdM program may be abandoned. This can result in declines in availability while the maintenance strategy reverts to more costly approaches such as reactive maintenance.

To mitigate this risk, MSD will continue to have a relationship with the PdM contractor to support the program when required.

Another pitfall to avoid is deciding that a PdM program can be done-on-the-cheap. For example, simpler, easy-to-learn PdM technology tools are acquired, but more sophisticated suites are never applied, even with contractor support. In such cases the diagnostic and long term analysis capabilities of advanced PdM software and the full potential of a comprehensive PdM program are not realized.

Or (as occurred at MSD) sophisticated and costly PdM tools are acquired and put in the hands of maintenance crew personnel who were given training they didn’t apply soon after the classes (so they lose the benefit of it), or had too few opportunities (in small plants with few assets) to learn on-the-job how to apply them or were transferred to other duties not including use of the tools upon which they had been trained.

In choosing a PdM mentoring organization it is important to assure the advanced technology hardware and software suites with which mentors are familiar (especially for vibration analysis and motor testing but less so for infra red thermography and ultrasonic analysis) are as close as possible to what will be acquired for use by the in-house team. This simplifies the mentoring process by limiting learning to the team, rather than both parties.

Consultation with the contractor currently providing PdM monitoring services may help in the quest to select the right vibration, infra red thermograph and other technology suites for the in-house program. As a general rule, monitoring contractors select for their own use packages that are the most productive and efficient in meeting both their needs as well as those of their customers. They may also be able to help by providing insight on how good high-end hardware and software vendors are at providing post-sale customer support, an important factor at all stages of development of an in-house PdM program.

Elements of a Plan That Can Be Used to Start, Create Major Change such as Bringing In-house and/or Convert to a Two-tiered (or any type of) PdM Program10 - The quote “Failure to plan is planning to fail,” is widely attributed to publisher, inventor, statesman and signer of the Declaration of Independence, Benjamin Franklin. The advice it provides is as applicable to a predictive maintenance program as any other important initiative. Below is a list of items that are recommended to be addressed in what at the Metropolitan Sewer District of Greater Cincinnati is called the Predictive Maintenance Master Plan, Within the body of the plan –

• Program Overview and How It Meets Maintenance Strategy Goals and Objectives
• Predictive Maintenance Team Description and Responsibilities
• Predictive Maintenance as Practiced by Others in the Organization (e.g., local maintenance crews)
• Predictive Maintenance Processes
  • Overall
  • Data Collection
  • Post Maintenance Testing
  • Baseline Data Collection
  • Predictive Maintenance Cost Avoidance Determination Process
• Predictive Technologies to Be Employed - Overview – Details in Annexes to the Plan
• Predictive Maintenance Report Management & Communications
  • Including Publicizing Program Accomplishments
• Key Performance Indicators – For Overall Maintenance Program and Its PdM Portion
• Predictive Maintenance (5 Year) Budgets for:
  • Analysis and Communications Computers, Software (including PDMMS) and Peripheral Equipment - Printers, Smart Phones, Tablets, etc., - As Required, Depending on Geographic Area and Number of Plants and Assets to Be Covered) 
  • PdM Hardware and Software Acquisition/Replacements/Upgrades
  • PdM Hardware Calibration and Repair
  • Training, Mentoring and Certification
  • Contractors Such as for Lubricant and Wear Particle Laboratory Services
  • Transportation (e.g., Dedicated Vehicles as Required)
  • Consumables

Annexes to the plan should be provided for each technology as well as for training and certification of each team member and any other participant(s) to be engaged in predictive maintenance. For example, the following annexes are provided in the MSD PdM Master Plan:

• Vibration Analysis
• Infra Red Thermography
• Ultrasonic Analysis
• Off-Line and On–Line Motor Testing
• Lubricant & Wear Particle Analysis
• Laser Alignment Tools
• Predictive Maintenance Management & Communications Software (PDMMS)
• Individual Training Plans for PdM Team Members
• Group Training Plans for Decentralized local Maintenance Crew Members

Annexes provide details for employment of technologies and goals and target dates for training courses and certification of PdM program participants employing the technologies for as far into the future as possible (2 to 5 years).

Accompanying the plan (but not necessarily included within it) should be an action item check list that includes all tasks needed to accomplish what’s in the plan for the next 3 to 6 months.

Depending upon the amount of effort involved to reach goals of the plan and obstacles or barriers encountered, the plan will have to be revised. In the first year this may be required quarterly or more often, the second year every six months and thereafter at least annually or with key personnel changes (e.g., training plan for new PdM team member or addition of a new technology).

Summary

A two-tiered PdM program applying multiple technologies has advantages for some organizations, especially for those serving multiple plants dispersed over a wide geographic area and/or where semi-autonomous maintenance crews are employed.

Consideration should be given in all types of PdM arrangements (team only, decentralized or two-tiered) to have PdM program management and communications separate from CMMS especially where “at risk” status of assets subject to PdM monitoring and KPI’s related to the program are important enough to be tracked.

Starting up from scratch or bringing in-house a PdM program being conducted by outside contractor(s) has benefits and many pitfalls that can be avoided by learning what’s in this presentation, given that it is done for the right reasons.

Developing a plan for what’s to be accomplished is as important for a predictive maintenance initiative as it is for any other “game-changing” program startup or major change.11

1 Hines, J. Wesley, Jesse, Stephen; Edmonson, Andrew; and Nower, Dan - “Study Shows Shaft Misalignment Reduces Bearing Life,” Maintenance Technology Magazine April 1999, pp 11-17

2 The benchmarking visit was hosted by John Butine, Manager of Field Operations Services Group and PdM team personnel of The Timken Company at their facilities in and around Canton, Ohio in August 2012.

3 The vehicle selected was TANGO by 24/7 Systems, Inc., of Knoxville, TN. For details see http://www.tf7.com

4 The vibration analysis and infra red thermography services, training, mentoring and certification support contractor is IVC Technologies of Lebanon, Ohio.

5 CMMS (MAXIMO Version 4) is being upgraded to MAXIMO Version 7 in 2013, with support from Brown and Caldwell.

6 Under the guidance of Richard (Doc) Palmer, P.E., CMRP of Richard Palmer & Associates, author of the Planning and Scheduling Handbook published by McGraw-Hill Inc.,

7 Under the direction of Anthony M. (Mac) Smith, P.E., of AMS Associates, Inc., with Tim Allen, CMRP, of Granite Reliability Group, LLC, a number of analyses have been completed in the past four (4) years, many recommendations from which are in process of being implemented.

8 The individual engaged was Jack R. Nicholas, Jr., P.E., CMRP an independent consultant and co-author of 10 books on maintenance subjects, including Predictive Maintenance Management published by Reliabilityweb.com

9 The CMMS upgrade from MAXIMO 4 to MAXIMO 7 is being supported by David Evered and other experts on MAXIMO from Brown and Caldwell.

10 See Nicholas, J. R. and Young, R.K text entitled Predictive Maintenance Management 3rd Edition (2007) published by Reliabilityweb.com

11Acknowledgement: Aside from the co-authors and others already mentioned in footnotes to this paper, the PdM program described would not have been possible without the active involvement and participation of the following people at various stages in its development to date:

From MSD of Greater Cincinnati - Andrew Bass, David Bargerstock, Charlie Becker, Christopher Bingham, Aaron Conley, Kevin Cunningham, Michael Duffy, Seth Featherston, Tom Goodman, Rick Hamant, Nancy Harrison, Jerry Hood, Rob Johnson, Pat Kane, Steven King, Samuel Klontz, Mark Koch, Will Leveridge, Doug Little, Logan Little, Jeremy McCleese, Tim McGrath, Richard Meade, Brent Merwin, Robert Metz, Zeno Perry, Robert Pfirrman, Andrew Sampson, Larry Scanlan, Glen Smith, Matthew Smith, Robert F. Smith, Daniel Soukup, Eric Stevens, hardworking staff at MSD, MSD Executive Director Tony Parrott, MSD Deputy Director Biju George, and Don Linn WWT Superintendent.

From outside MSD: Pete Epperson, Steve Smith and trainers from IVC Technologies; Sam Paske from Brown & Caldwell and now with CH2M Hill; John Fortin of C2HM Hill; Jaclyn Gandee, subcontractor to Brown & Caldwell; Forrest Pardue from 24/7 Systems, inc.