Participant Impact

EMMS data model v5.2

Participants wanting to receive the new and updated Data Model The definition of the interface to participants of data published by AEMO for gas or electricity. A database conforming to the Data Model can contain a local copy of all current participant-specific data recorded in the main database. The Data Model includes database tables, indexes, and primary keys. information in their Data Interchange A set of cooperating applications used to replicate data between AEMO's energy market systems and a participant's DBMS conforming to the MMS Data Model. environments must upgrade to the latest version of the Data Model v5.2.

Participant systems incorrectly configured and not compliant with the Baseline Assumptions may suffer data loss.

Changes to DUDETAILSUMMARY

Be aware of changes to the Modified table: DUDETAILSUMMARY table.

Increasing data volumes

As a result of the following two factors, the content in the NEXT_DAY_OFFER_ENERGY report has exceeded 4 million rows and the content in the NEXT_DAY_OFFER_FCAS Frequency Controlled Ancillary Services. A service that balances, over short intervals (shorter than the dispatch interval), the power supplied by generating units and the power consumed by loads. report has exceeded 20 million rows (vary from day to day depending on bidding activity):

  1. The introduction of 5MS into the NEM moved the Trading Interval from 30-minute to 5-minutes, increasing by sixfold the data supplied to AEMO.
  2. The continued growth of renewable Plant registered in the NEM as Semi-scheduled or Scheduled has increased the number of DUIDs.

The continued growth of these reports is presenting the following issues to participant systems:

Fast frequency response

The NEM currently includes a total of 9 services: One Energy The unit must be a generator for the service type: ‘energy’. Loads cannot bid for mandatory restrictions capacity. and 8 FCAS. The introduction of Fast Frequency Response The information returned by an API after a request is made. Responses are usually in JSON or XML format. (FFR Fast Frequency Response) introduces 2 more services into the NEM, exasperating the emerging scale issue.

Sparse Model

An analysis of the data by AEMO Australian Energy Market Operator identified adopting a sparse data model, where repeated data is supressed, can potentially result in a 95% reduction in data volumes in these reports.

To ensure data volumes remain within acceptable limits, AEMO is moving to a sparse representation for Energy and FCAS Bids and Offers. This is necessary to allow us to continue meeting our Market data reporting obligations.

The sparse model stores the period row where offered values for period (n) differ from period (n-1). Yellow is persisted data.

Offer

P1

P2

P3

P4

P5

P6

P7

P8

First offer

5

5

5

10

10

5

5

5

Next offer

5

10

5

5

10

5

5

5

For full details about the Model and AEMO analysis, see Increasing data volumes.

Sparse Model update

Most participants agreed with the opt-in sparse model, so we have delivered the proposed functionality in pre-production for the following NEXT_DAY_OFFER_* only reports:

  • NEXT_DAY_OFFER_ENERGY
  • NEXT_DAY_OFFER_FCAS

All participants subscribed to the current version of the above files are now moved to the Legacy versions:

  • NEXT_DAY_OFFER_ENERGY_LEGACY
  • NEXT_DAY_OFFER_FCAS_LEGACY

I want to receive the same data as previously

To receive the same data, remain on the Legacy versions of the above files.

I want to receive the sparse model

To opt-in to receive the sparse data, subscribe to the latest versions of the above files.

Forward plan for sparse offers

Forward plan for sparse offers

For Data Model v5.2, the Sparse Mode is delivered as opt-in only, in a future release it will become the standard representation of NEXT_DAY_OFFER_* data where functional changes are identified for this data set.

We advise participants to begin transitioning their downstream dependencies to interpret sparse offer data now in readiness for this future change.

Future consideration

In a future release, we are considering extending this same functionality for confidential offer reporting and offer submission.

Upgrading your data model

You can upgrade your pre-production or production Data Model environments once you receive the Data Model scripts. Applying the scripts sets up the new Data Model structure on your local database. You receive the same data until the new versions of fields, files, and reports are released into pre-production or production and you update your subscriptions.

Depending on the project, some tables and reports populate at different stages. Unpopulated tables and reports either remain empty or display null values. For details about individual project release dates and when the data populates, see Projects.

For help, see:

Data subscription

Legacy files

On the Release Dates, AEMO moves participants subscribed to existing files to the Legacy version. After you have upgraded to v5.2, subscribe to the current files in Data Subscription A web application in the Markets Portal used to subscribe to Data Interchange files and to obtain the latest MMS Data Model. These files are typically csv files containing public data and participant’s private data. AEMO recommends participant use of the Data Interchange suite of products, supplied by AEMO, to automate receiving and storage of the files to their systems. AEMO provides working sample software for FTP access (Participant Batcher) to MarketNet and working sample software for Data Interchange.. For help, see Subscribe to Files.

Auto-subscription

Existing participants are automatically subscribed to the following new files and don’t need to do anything:

File ID

Table

AGGREGATE_DISPATCH_GROUP

AGGREGATE_DISPATCH_GROUP

ADG_DETAIL

Software

Data Model Manager

This Release has an upgrade to the Data Model Manager (pdrDataModelManager), version 1.1.0.

Data subscription and pdrLoader

EMMS – Release Schedule – Data Interchange – April 2023Details of changes to the Data Interchange products: Data Subscription and pdrLoader Software responsible for loading files to and from participant's local Data Interchange folders to the participant's DBMS. It can be installed in any environment supporting Java.