Data Interchange Architecture

About Data Interchange

Data Interchange is a set of cooperating applications to replicate data between AEMO’s Energy Market Systems and participants' local DBMS conforming to the MMS Data Model, including management and monitoring.

Data Interchange has two core functions:

  1. AEMO-side reporting applications that generate structured .CSV files into the participant file server.
  2. Participant-side software to replicate data from the participant file server to participants' local DBMS.

Data Interchange components

Data Interchange components describes the components required for a full installation of Data Interchange.

Table 19 Data Interchange components

Component

Description

MMS Data Model (MMSDM) or Gas Data Model (GASDM)

Establishes the target tables in a participant's DBMS conforming to the MMS Data Model.

Participant Data Replication Batcher (pdrBatcher)

Responsible for transferring files to and from AEMO's participant file server.

Participant Data Replication Loader (pdrLoader)

Responsible for loading files from participant's local file repository to the participant's DBMS.

Participant Data Replication Monitor (pdrMonitor)

Manages the operation of the overall solution using a browser interface.

Data Subscription

The web application in the energy market systems web portal allowing registered participants to:

  • Obtain the latest MMS Data Model scripts for building and maintaining their DBMS
  • Subscribe and unsubscribe to the report files available in the participant file server
  • View the entire list of available files.

    Note: AEMO recommends subscribing to all available files and selectively loading them into your DBMS as required.

Data Delivery

The web application in the energy market systems web portal allowing registered participants to:

  • View the Data Interchange Status Dashboard
  • Make a file delivery query
  • Re-request files
  • View your organisation's data sharing arrangements.

Performance Monitor

The web application in the energy market systems web portal allowing registered participants to view a graphical interface of the current Data Interchange Performance Monitor environment run and managed by AEMO. Participants not receiving data, or experiencing data delays can contrast the data delivery performance of their Data Interchange environment to AEMO's Data Interchange performance environment.

Data Interchange bundles

All the components necessary for a new installation of Data Interchange are bundled together into compressed files; each bundle is for a specific DBMS and version of the MMS Data Model.

The Data Interchange bundles are available from:

  • The Data Interchange web page on AEMO's website.
  • The Releases directory: FTP to 146.178.211.25 > Data Interchange.

Software and patch scripts

Data model upgrade

Data Interchange software, Data Model upgrade and patch scripts are available from the Participant File Server. FTP to 146.178.211.25 > Releases >:

  • Gas Data Model
  • Electricity Data Model
  • Data Interchange
  • pdrBatcher
  • pdrLoader
  • pdrMonitor

For help upgrading, see Upgrading a DI Environment.

Data interchange full install

To set up a new standard Data Interchange environment, you can download the bundles for your DBMS and follow the steps for Setting Up a Standard DI Environment.

Standard Data Interchange implementation

AEMO recommend setting up your first Data Interchange environment in pre-production. Once you have a successful pre-production installation (used for testing), you can replicate it in production to receive live market data. Be careful to always keep the pre-production and production data entirely separate.

You can only have one DI implementation connected to one participant file server environment. For example, you can have your pre-production environment connected to the participant file server pre-production environment and your production environment connected to the participant file server production environment.

standard Data Interchange implementation describes the standard Data Interchange implementation which is a single pdrLoader instance for each database.

Figure 11 standard Data Interchange implementation

Requirements

Successfully implementing and managing Data Interchange requires good IT skills, including networking, database management, batch file management, and disaster recovery.

To use Data Interchange requires:

  • A user name, password, and user rights, provided by your company's participant administrator (PA), providing access to the Data Interchange web applications in the energy market systems web portal, User rights access.
  • A DBMS supporting the MMS Data Model. The MMS Data Model supports versions of Oracle and Microsoft SQL Server.
  • Java runtime engine suitable for the target DBMS. The distribution file from AEMO contains supported JDBC drivers for Oracle and SQL Server.
  • The Replication Manager software runs on Windows OS only, other Data Interchange components run on both Windows and Unix-like operating systems.
  • For accessing the energy market systems web portal, Microsoft Internet Explorer version 7 or later, although the recommended version is Microsoft Internet Explorer 8.
  • A user name, password, and user rights, provided by your company's participant administrator (PA), providing access to the Data Interchange web applications in the energy market systems web portal, User rights access.
  • A DBMS supporting the MMS Data Model. The MMS Data Model supports versions of Oracle and Microsoft SQL Server.
  • Java runtime engine suitable for the target DBMS. The distribution file from AEMO contains supported JDBC drivers for Oracle and SQL Server.
  • The Replication Manager software runs on Windows OS only, other Data Interchange components run on both Windows and Unix-like operating systems.
  • For accessing the energy market systems web portal, Microsoft Internet Explorer version 7 or later, although the recommended version is Microsoft Internet Explorer 8.

User rights access

Your company's participant administrator (PA) grants you permission to use the Data Interchange components within the AEMO Market Portal. The entities required for access to each component are listed below. Where a participant user has user rights assigned by more than one participant, they interactively choose the participant they represent, using the Set Participant option.

For more details about participant administration and user rights access, see Guide to User Rights Management.

Data Subscription

  • EMMS - Data Subscription - Maintain and View Data
  • EMMS - Data Subscription - View Data

Data Delivery

  • EMMS - Data Interchange

Performance Monitor

  • EMMS - Data Interchange - Performance Monitor

Can the PDR software run on a UNIX platform?

Yes, the software is written in Java which is a platform independent language. Shell script wrappers are provided for both pdrBatcher and the pdrLoader that may need to be customised to the particular flavour of Unix that is being used.

Can pdrBatcher and pdrLoader run on the same server?

Yes, they can also run on the database server itself. They are provided as separate products to allow for more complex and secure networking configurations.

How can I set up FTP access to AEMO systems?

To enable FTP access using participant ID(s), participants contact their company's IT Security contact or participant administrator (PA) to obtain the credentials issued by AEMO.

How can I identify the version number of AEMO Data Interchange applications?

Check the log file to identify version. Restarting the application also shows the version number.

What is the recommended FTP mode to run the pdrBatcher?

AEMO recommends running FTP in PASSIVE mode at all times, as per the default configuration for the pdrBatcher.

Using ACTIVE FTP mode may work, however, it is less reliable because there are potential issues with the FTP port number being decided at the participant end and hence collisions can occur on the AEMO server. ACTIVE FTP mode may not work with certain networking configurations.

Can I run multiple instances of pdrLoader against a single database?

AEMO has not specifically tested this scenario and recommends using the standard set-up described in the Guide to Setting up a standard Data Interchange environment. A standard Data Interchange implementation is a single pdrLoader instance for each database, Standard Data Interchange implementation.

Can I run multiple instances of pdrBatcher or pdrLoader out of a single installation directory?

Yes, but each must have its own set of working directories and log file.

Setting up multiple instance of the pdrBatcher and pdrLoader out of a single installation directory requires sophisticated technical knowledge of Data Interchange. AEMO recommends setting up a standard Data Interchange environment to verify it is working before attempting to run multiple instances from a single installation directory, Standard Data Interchange implementation.

Use the config and log4j command line parameters when invoking the application from the batch or shell script. These same parameters can be adjusted in the Windows service wrapper.

This is useful when setting up a disaster recovery (DR) pdrBatcher that has a main configuration to point to the production pdrBatcher to extract report files for the DR environment. In this case, setup a separate invocation script pre-configured that launches the DR pdrBatcher against the AEMO production system to bring up the DR site for production duties quickly.

Can I send the logging output of pdrBatcher and pdrLoader to the same log file?

The two processes should use separate log files, particularly when using the DailyRollingFile appender in log4j.

How do I set up a disaster recovery environment using PDR?

To setup a disaster recovery (DR) environment:

Setup the production environment to have a pdrBatcher configuration that tees outgoing directories (Reports and ReportsTrickle) to two separate staging directories. The production pdrLoader consumes files from one of these directories into your production database. The DR pdrBatcher then points to the alternate set of staging directories fed by the production pdrBatcher and feeds these files to the DR pdrLoader.

An equivalent return path from the DR pdrLoader Request directory must feed files back to the production pdrLoader Request directory so these can be chain fed back to AEMO's systems, for help see, the Participant Data Replication Batcher User Guide.

The most critical configuration aspect is that each pdrLoader must have a unique instance_identifier defined in the properties configuration file.

What is the difference between trickle feed and normal CSV feed?

The trickle feed provides a mechanism for a low priority catch-up, and fills in data gaps in historical data when there is idle time. The CSV feed is the real time data feed and has priority over any backfill operations.

Can I retrieve a historical file that I was not subscribed to?

No, AEMO only keeps a record of reports that each Participant ID was subscribed to at the time that a report was generated. AEMO recommends subscribing to all available files and selectively loading them into your DBMS as required.

Participants can load public historical data from AEMO's Market Data NEMweb. For more information, see Populating historical data to data model tables.

How long is information retained on the AEMO online archive?

AEMO has sized hardware to retain approximately six month's worth of files for all participants.