File Auto-recovery

What is auto-recovery

Auto-recovery is functionality built into the pdrLoader that provides an assurance of guaranteed receipt and commit to the participant DBMS of all published data to a participant.

How does auto-recovery work?

The pdrLoader periodically issues a manifest request. This only occurs when the loader is idle and there are no files waiting to be loaded. The manifest request is a small text file that contains a date-time stamp that is the end date-time of the last transaction manifest received. If it is a first time manifest, then the current date-time is used.

The pdrBatcher transfers this request to AEMO's systems, which process the request file and provide an acknowledgement. The acknowledgement file is in the standard AEMO CSV format and is processed into the database to update the STATUS and STATUS_DATE columns of the request in the PDR_REQUEST_LOG table. This update only applies if the instance_identifier in the response file matches the one defined in the pdrLoader.properties file. This is how the system ensures that data coming out of AEMO's systems find its way to the correct instance.

The acknowledgement of a manifest request means the report request is logged in AEMO's systems and will be delivered in a timeframe that does have dependency on the loading of AEMO's systems. Generally, it will be within a minute of the request.

The manifest report is a list of all files that AEMO has a record of delivering to the participant directory forward of the request date-time. The returned manifest does contain an overlap, ensuring there are no boundary conditions around the request date-time. This data (assuming again the instance identifier is appropriate) is then loaded into the PDR_MANIFEST_LOG table and set to a "RECONCILE" status. Additionally, the MANIFEST_END_DATETIME column in the PDR_REQUEST_LOG associated with the request is updated so the system has a record of the timestamp that AEMO's systems have provided data for. This end time becomes the request date-time of the next manifest request.

The initial RECONCILE state of data in PDR_MANIFEST_LOG indicates that when the reconciliation process runs, it must check all rows to see if there is a matching entry in the PDR_REPORT_AUDIT table with a SUCCESS status indicating the file is loaded. Files are identified by EVENT_QUEUE_ID, FILE_ID, CONFIDENTIALITY columns. All rows having matching records in PDR_REPORT_AUDIT are then deleted from this table and under normal circumstances this leaves no rows. If there are any rows left, these rows are set to a MISSING status, making them a candidate for re-requesting.

When the re-request process runs, it extracts all rows (up to a configurable limit of 100) with a MISSING status and then generates an archive request. An archive request is a list of files that the participant system is requesting AEMO's systems to resupply using the trickle feed. A response to an archive request is on a per row and per file basis with AEMO applying a threshold to how many re-requests are accepted in any single archive request.

The archive response file is a standard AEMO CSV format file that updates the rows in the PDR_MANIFEST_LOG to indicate a STATUS of re-requested and the archive_request_id.

When the files are loaded, the rows in the PDR_MANIFEST_LOG are deleted the next time the reconciliation process runs, gradually clearing the gaps.

Independently of this process, a timeout thread is used to manage the scenario where a request is acknowledged by AEMO's systems but no corresponding response is received. This ensures that if AEMO's systems do not respond appropriately, the participant system can return the rows to a state where it can repeat the request and ensure all data gaps are cleared.

For help with the CSV format, see Guide to AEMO CSV Data Format Standard.

Missing data not recovered

Check and action the following:

  1. File automated recovery set to true:
  • Set pdrLoader.properties file for parameter file_automated_recovery=true.
  1. Recent changes to loader parameters?
  • Restart pdrLoader to make the changes effective.
  1. When was the last MANIFEST requested and its status?
  2. Select request_date, manifest_start_datetime, status from pdr_request_log

    Where request_date= (Select max(request_date) from pdr_request_log where request_type=’MANIFEST’)
  3. Restart the pdrLoader.
  1. When was the file subscribed?
  • Missing data cannot be recovered for periods prior to the file subscription date.
  1. RESPONSE set to ACTIVE so the pdrLoader can process Response files?
  • pdrMonitor>main page>RESPONSE set to ACTIVE
  1. MANIFEST set to ACTIVE so the pdrLoader can process the MANIFEST reports?
  • pdrMonitor>main page>MANIFEST set to ACTIVE.
  1. Are the acknowledgements files receive by the pdrLoader?
  • pdrMonitor>main page>last update for record type RESPONSE.
  1. Are the MANIFEST reports received by the pdrLoader?
  • pdrMonitor>main page>last update for record type MANIFEST.
  1. MANIFEST request files are sent to AEMO?
  2. Check pdr>Loader>Request directory for any build up of MANIFEST request files.
  3. Resolve any pdrBatcher issues for not sending the requests.

PDR MANIFEST log table keeps growing

Check and action the following:

  1. The \\loader\\ReportsTrickle directory is available for the pdrLoader:
  2. Analyse the pdrLoader thread configuration/masks for possible exclusions from the ReportTrickle directory.
  3. Adjust the pdrLoader thread configs if required.
  4. The batcher downloads the \\PARTICIPANT_ID\\IMPORT\\REPORTS\\CSVTrickle:
  5. Analyse the pdrBatcher thread configuration/masks for possible exclusions from the CSVTrickle directory.
  6. If a single batcher is feeding multiple pdrLoaders, ensure the trickle feed is teed off to all loader instances.
  7. Adjust batcher thread configurations if required.

REQUEST not acknowledged

Check and action the following:

  1. A batcher thread is configured to send the REQUEST pdrLoader directory files to \\PARTICIPANT_ID\\EXPORT\\DATA_INTERCHANGE:
  • Configure pdrBatcher thread.
  1. Any REQUEST file build up in the pdrLoader request directory:
  2. file_request_dir=c:\\pdr\\loader\\Request. Investigate for the pdrBatcher not sending the requests to AEMO.
  1. RESPONSE set to ACTIVE so that Response files are processed by the pdrLoader:
  • pdrMonitor>main page>RESPONSE set to ACTIVE.

MANIFEST request not generated

Check and action the following:

  1. File automated recovery is set to true:
  1. Check the pdrLoader.properties file for parameter file_automated_recovery=true.
  2. Set the Parameter file_automated_recovery=true.
  3. The participant_identifier is correctly set in the pdrLoader.properties.
  4. The instance_identifier is unique across all Loader instances.
  5. A batcher thread is configured to receive MANIFEST reports from \\PARTICIPANT_ID\\IMPORT\\ACKNOWLEDGMENTS and deliver them to the pdrLoader reports directory.
  6. A batcher thread is configured to send the REQUEST to \\PARTICIPANT_ID\\EXPORT\\DATA_INTERCHANGE.

pdrLoader is running but has not requested a MANIFEST

The pdrLoader only requests a manifest when there are no .CSV files waiting to be loaded. If you have a significant data backlog, no manifest operations occur until all the pending data is loaded. Check and action the following:

  • Ensure the automated recovery feature is enabled in the pdrLoader.properties file.

MANIFEST timed out

Check and action the following:

  • The timeout to receive ACK/MANIFEST report:
  • The pdrLoader.properties  file_load_request_timeout (default=120 sec). Increase the pdrLoader file_load_request_timeout value if necessary.

LEGACY files keep appearing in MANIFEST

Check and action the following:

  • If running the latest data model version, unsubscribe from LEGACY files:
  1. In the Data Subscription interface in the energy market systems web portal unsubscribe from LEGACY files.
  2. In pdrMonitor, set LEGACY files to ABANDONED and monitor replication to avoid re-requests.