Sparse Data Model
Background
The content in the NEXT_DAY_OFFER_ENERGY report has exceeded 4 million rows and the content in the NEXT_DAY_OFFER_FCAS Frequency Controlled Ancillary Services. A service that balances, over short intervals (shorter than the dispatch interval), the power supplied by generating units and the power consumed by loads. report has exceeded 20 million rows (varying from daily Energy Rules Terms depending on bidding activity), as a result of the following:
- The introduction of 5MS into the NEM moved the Trading Interval from 30-minutes to 5-minutes, increasing by sixfold the data supplied to AEMO.
- The continued growth of renewable Plant registered in the NEM as a Semi-scheduled or Scheduled Generating Unit has increased the number of DUIDs. Especially where many of these Plant are using automated bidding solutions which offer a new capacity or trading position into the market each 5 minutes.
The continued growth of these reports is presenting the following immediate issues to some participant systems:
- Time to download the content over the limited bandwidth available to MarketNet AEMO's private VPN for Registered Participants. A network of more than 300 registered participants in the National Electricity Market, including market generators, transmission network service providers, distribution network service providers, and market customers..
- Time to process the file into the Participant Data Model The definition of the interface to participants of data published by AEMO for gas or electricity. A database conforming to the Data Model can contain a local copy of all current participant-specific data recorded in the main database. The Data Model includes database tables, indexes, and primary keys. instance.
Future issues
The NEM Reform program will also drive increased volumes in these reports. The NEM currently includes a total of 9 services: One Energy The unit must be a generator for the service type: ‘energy’. Loads cannot bid for mandatory restrictions capacity. and 8 FCAS.
The introduction of Fast Frequency Response The information returned by an API after a request is made. Responses are usually in JSON or XML format. (FFR Fast Frequency Response) introduces 2 more services into the NEM, exacerbating the emerging scale issue of the NEXT_DAY_OFFER_FCAS report, perhaps by as much as 20%.
What is proposed?
An analysis of the data by AEMO Australian Energy Market Operator identified adopting a Sparse Data Model, where repeated data is supressed, can potentially result in up to a 95% reduction in data volumes in these reports.
The Sparse Data Model stores the period row, where offered values for period (n) differ from period (n-1). Yellow is persisted data.
|
Trading interval |
Example offer 1 |
Example offer 2 |
|---|---|---|
|
1 |
5 |
5 |
|
2 |
5 |
10 |
|
3 |
5 |
5 |
|
4 |
10 |
5 |
|
5 |
10 |
10 |
|
6 |
5 |
5 |
|
7 |
5 |
5 |
|
8 |
5 |
5 |
The interpretation of a Sparse Data Model offer representation is where an offer value for period (n) does not exist, then the value for the prior period is carried forward. In the above example in the Example offer 1 column both Trading Intervals 2 and 3 inherit the value 5 from Trading Interval See Relevant Rules or Procedures 1.
In the Data Model v5.2 release May 2023, AEMO proposed the concept of a Sparse Data Model delivery for the NEXT_DAY_OFFER_* reports and an eventual discontinuation of the fully qualified reports on the basis they are not seen as being sustainable in the medium to longer term.
We received feedback from participants in the Market Systems User Group (MSUG Market Systems User Group) requesting some adjustments to the originally proposed design to accommodate a longer transition window, so we separated the functionality from the delivery of the Data Model v5.2 release.
Baseline assumptions
For more details about the Data Interchange A set of cooperating applications used to replicate data between AEMO's energy market systems and a participant's DBMS conforming to the MMS Data Model. suite and the baseline Energy Rules Terms assumptions, see Baseline Assumptions