Pacific Gas and Electric (PG&E, SanFrancisco, Calif.) is the largest utility in the United States, and even after its recent downsizing, still employs over 25,000 people. Its "Line 400'' pipeline brings natural gas from the Oregon border to the San Francisco Bay area via four compressor stations stretched along the state. Each station's primary function is to terminate one section of the pipeline and recompress the gas with 15,000 hp turbines before sending it to the next station. This segments the system failure risk and providing monitoring and safety shutoff points.

  These four compressor stations are usually unmanned and must be remotely monitored and controlled by several layers of telemetric supervisory control and data acquisition (SCADA). Historically, only major station functions such as station status, pipeline pressure, and gas flow rates were monitored and controlled by PG&E's corporate-wide gas SCADA system. But in the early 1990s as available control technology's ability to monitor an increasing number of components improved, PG&E went in search of higher station efficiencies through condition-based maintenance (CBM). CBM was intended to replace the often unnecessary and costly scheduled maintenance of station equipment with "service-as-needed." Remote access to condition-critical data such as turbine hp output, fuel consumption, duty cycles, and shaft vibration became a must.

Defining the problem

..Creating electronic windows on station equipment using PG&E's corporate-wide SCADA equipment would have required significant expense in hardware upgrades, software revisions, and training for the operations department to effectively and safely tap into its 300-plus remote terminal unit (RTU) control net. After much discussion, PG&E's engineering and operations departments jointly decided that a new, smaller telemetric SCADA system should parallel the corporate system, allowing user-friendly access to CBM data either remotely or locally.

  For use with local hosts and human-machine interfaces (HMIs), PG&E chose the Monitor, an Apple Macintosh-based system produced by Monico Systems (Austin, Tex.). The Mac-based system was chosen because it allowed field technicians with minimal computer skills to easily handle both basic programming and a more advanced data trending program.

The fun begins

..Because of varying yearly budgets, evolving technologies, and differing philosophies of PG&E's control engineers over the years, none of the compressor stations was automated with exactly the same control architecture. Also, very few of the local control and monitoring devices at each station spoke the same language or spoke at the same speed.

  At the time of installation, the Monitor system could only communicate with about one-third of the equipment required on the network. Protocol requirements ranged from Modbus to obscure dialects spoken by GE Speedtronic (compressor control) and other esoteric control devices embedded in pipeline-specific equipment. Control "dialects" included Modbus RTU, Data Highway, Modbus 1.0, GE Data Dump, HART, and some even less mainstream communication protocols.

  Additionally, the Monitor system could only handle one transmission rate at a time. It had to be reconfigured whenever devices with different rates had to be polled or signaled. Baud requirements also ran the gamut from 1,2K to 38.4K bits/sec. In addition, there had been little consistency in device addressing. Even within the same site, various controllers had been assigned duplicate address codes, which made networking impossible.

  The new system only provided 12-bit data resolution that made floating point applications, requiring 16- or 32-bit resolution, impossible. The combination of high precision logging requirements and a hodgepodge of duplicate and uncoordinated device address codes left the little Macintosh icon's faces on the Monitor's screen with their eyes crossed and the system jammed.

  Ultimately, PG&E's controls group had to get this control-device polyglot integrated into a multilingual, telemetric SCADA system that would:

• Provide the equipment performance data necessary for long-term trending and analysis,
• Provide the equipment-status data necessary to keep their equipment running using CBM,
• Be easy for field technicians to use and maintain, and
• Be remotely accessible by all necessary departments within PG&E.

Developing the solution

..DST Controls, a control systems integrator,was brought in to identify the controllers and field devices to be networked, categorize equipment communications requirements, and define the data-sets to be extracted from each device. DST then handed the resulting list of driver requirements to Diamond Process Applications (DPA, Santa Cruz, Calif.) for development.

  The ultimate solution, however, was to take form in a protocol conversion device that would be easier and more versatile to use than the port expanders and translators that DST Controls has used previously. The solution had to allow PG&E's new SCADA system to communicate with monitoring and control devices, which could not communicate, because of protocol and/or communication rate incompatibilities. It would also act as a "data concentrator," downline from the Mac-based host, that would extract data from the "mix-matched" field devices and store the data until polled by the host.

  Initially, DPA took previously developed co-driver code sets and linked them with core code, creating a standalone system on a 486 single-board computer. While PG&E's custom driver library was being created, DST Controls defined how the acquired data would be displayed and archived, made up and installed the custom communications cables, and produced system documentation and operation and maintenance manuals.

  An application-mirroring program also was added. This allowed the new data concentrator/protocol converter to be virtually superimposed on any PC in the system, via a modem, allowing troubleshooting and configuring to be done remotely. The turnkey system was then placed in an industrially hardened 486 PC enclosure and installed at PG&E's Delevan, Calif. compressor station.

  The protocol converter/data concentrator that first arrived at Delevan turned out to bethe prototype for what is now DPA's "plug and play" IB-8000 Industrial Communications Bridge. The unit is a multiprotocol converter/data concentrator for a wide range of industrial communications protocols used by various manufacturers of DCS, SCADA, RTU, PLC, single-loop controllers, monitoring equipment, andfield I/O devices.

  The device allows protocol conversion from an uncommon to a common one, thereby allowing data transfer to a SCADA system whichdoes not have a driver for that protocol. When initially installed, the bridge allowed up to32 RS-232/485 serial ports with two network adapters--Ethernet and Arcnet--to be supported simultaneously.

  Other control functions PG&E required included the ability to concentrate data from multiple devices and have those data available to the requesting station. In PG&E's case, this meant interfacing 30 RS-232 connections to a single port on a SCADA system (the Macintosh computer) and interfacing serial devices to a network using Ethernet or Arcnet and TCP/IP or Netbus. This capability allows user programs on a Unix or Microsoft Windows NT node to access plant-floor data using standard practice (socket calls).

  PG&E's inventory of controllers includes some "less than mainstream" equipment that also had to be part of the mini-telemetric network. Although the original SCADA package arrived at PG&E already supporting the station's Allen-Bradley, GE Fanuc, and Modicon PLCs, two additional days were required to get stragglers online.

  Aspects of the data concentrator that PG&E found especially attractive were:

• Ability to run on an inexpensive PC
• Data manipulation ability, which allows rescaling of analog data to meet the Monitor's display limitations
• Support for 50 serial ports
• Ethernet capability for TCP/IP or other protocols
• Ready acceptance of new drivers by the core program
• System scalability

  The system has been running for over two years. At one compressor station it monitors over 500 data points from 11 different devices on five serial ports. The functions monitored include:

• Turbine speed and hp output
• Gas pressure, temperature, and flow rate
• Control valve positions
• Duty cycles
• Gas detection system status
• Uninterruptible power supply status
• Fuel gas pressure and consumption rate
• Turbine bearing temperatures and gap readings
• Gas cooler inlet/outlet temperatures
• Gas filter differential pressure

  Statistical analysis has shown that in the two years of operation, more than 13 million data transactions (over 1,400 million bytes) were processed with only 33 errors.

  The capabilities of this mini-telemetric SCADA system, and its protocol conversion/data concentration kernel, united the numerous and disparate devices that bring natural gas to California. They also demonstrate that the innovative integration of new technology can allow preparation for the future while providing user-friendly systems necessary to keep the gas lines flowing now.

George Gaebler is a Gas Engineer at Pacific Gas and Electric in San Francisco, Calif.

Read Hayward is vice president, operations, of engineering at DST Controls Inc., Benicia, Calif.

The soft heart of the system

..Software is key to the functionality of the IB-8000 Industrial Communications Bridge, used by PG&E in the enhanced SCADA system. It is written in Microsoft C with low-level OEM libraries purchased with the hardware in certain cases. There is a core program that reads a configuration file on startup. All serial communications from each port are buffered, i.e., all messages coming in are put in a buffer--one per port--by an interrupt-driven I/O routine. All outgoing messages are also placed in the buffer by the driver and are sent by an interrupt-driven output routine.

  The core program then calls each driver in turn in a tight idle loop at least once every millisecond, and each driver handles as many serial ports as have been allocated to it by the configuration file. Each driver sends and receives messages out of every port simultaneously.

  Part of the core program's responsibility is to act as an human-machine interface (HMI), allowing the operator to view both performance and error statistics. The IB-8000's software was developed to run on standard 386 or higher PC hardware for several reasons:

• Ready availability of communications libraries and related software
• Availability of PC-oriented communication hardware
• Low development costs (no embedded debuggers or in-circuit emulators are required)
• "Open" hardware requirements
• Ease of troubleshooting

Trying something new

..Modbus port expanders, which were looked at originally, have been around for a long while. Their role is to allow multiple masters to access a single or multiple slave(s). A port expander is required because Modbus (and many other protocols) usually allows only a single master that requests data from all slave devices, so there was no confusion on the network as to which device can talk, and when. Slaves only respond to master requests.

  This works well except for when two or more masters request data from any PLC almost simultaneously. Since only one transaction can occur on the network at a time, all the other masters are put on hold until the first transaction is complete. This limits the bandwidth to one message at a time (if all the slaves are on a single line). Spreading the slaves over multiple ports eases the situation but does not solve it, since two SCADA systems can still request data from the same PLC simultaneously.

  This situation degenerates rapidly when one of the slaves fails. Since all requests for that slave are put on hold until various parts of the system time out, there is a chain reaction that backs up all the other SCADA systems. Even worse because of their time-outs, the SCADA systems have a difficult time trying to establish which slave has failed, or they can even be lulled into thinking an off-line slave is still online.

  The data concentrator approach eliminates this situation. With poll-block caching, when the request from the master arrives, the bridge first checks to see if there are data from the particular slave station already in a buffer. If yes, and the data are "fresh" enough, then the bridge responds immediately with these data. This saves the time delay involved with polling the slave station.

  If a buffer does not exist, one is automatically created. A poll will take place immediately to bring in data, which is returned to the master station. Data remain in the buffer for possible future requests. Buffers created in this manner are also polled on regular intervals in order to ensure that their data are as recent as possible. If the data in a buffer is too old to be reliable, a poll request is immediately generated. If this fails, then the system does not respond to the master request at all, indicating to the master that the particular slave station has gone off-line.

  If a buffer is not accessed by a master within a given time interval, then the buffer is removed from the poll list. The bridge becomes a slave to the SCADA system and becomes a master to the PLCs while buffering the data internally and effectively de-coupling the two sides.

The accompanying schematic illustrates the Monitor miniSCADA system implemented for data collection that facilitates compressor station monitoring.