File Issues

Turning off a file data feed?

To turn off a file feed, either:

  1. Set the IS_ACTIVE flag to false in the PDR_REPORT_TYPE_CONFIG table for the specific file.
  • If you set the file to inactive, the file is still generated and downloaded but its content is not loaded to your Electricity Data Model. In this case, AEMO maintains a copy of the file in its online archive so you can re-request it for up to six months. If you are not subscribed to a file it is not possible to re-request it at a later date.
  1. Unsubscribe from the file using the Data Subscription web interface.
  2. If you unsubscribe from a file it is no longer delivered to your CSV Reports directory on the AEMO file server and it is not possible to retrieve it at a later date.

AEMO recommends option 1 unless there is a specific reason to avoid the additional data download.

Manually re-requesting a file

To re-request a file:

  1. Using the pdrMonitor, double-click the Electricity Data Model package on the front interface to see the logs form.
  2. Select a block of reports and click the green re-request arrows on the toolbar.

Files in Error directory

Files in the error directory are unsuccessful loading attempts by the pdrLoader. Check and action the following:

  1. Check pdrLoader Logs for relevant error and resolve.
  2. Reprocess the file(s) (if file_load_automated_recovery is not set to true) or let the automatic recovery to re-request.
  3. Clean the directory contents periodically if auto-recovery is enabled.

pdrBatcher not receiving a particular file type

Check and action the following:

  1. Is this file type subscribed?
  • Use the Data Subscription web interface to verify that the related file is subscribed to. Select Subscribe to Files and then filter by Collection and Package.
  1. Is there a valid pdrBatcher thread configured for this file type?
  2. Verify the pdrBatcher.properties for this file type. File download using a specific thread? OR using a catch all thread(s)?
  3. Check the correct file mask is applied for the batcher thread.
  4. pdrBatcher thread count matches with the number of threads configured
  • Adjust the batcher_threads property in the pdrBatcher.properties.
  1. pdrBatcher logs for any errors:
  • Check Pdr>Batcher>Log>pdr.log and explore reasons for any batcher errors.
  1. Dead thread observed in the pdrBatcher log?
  • Restart the pdrBatcher to recover the dead thread.
  1. The files are produced at source:
  • FTP to AEMO and check the source directory for this file type.
  1. Check only a single pdrBatcher instance is reading the AEMO source:
  • If a secondary pdrBatcher exists, configure it to use an alternative source for example, a primary batcher copying to multiple directories.

pdrBatcher not receiving any files

Check and action the following:

  1. Timestamp is current on heartbeat_download.out file in pdrBatcher log directory.
  1. pdrBatcher logs for any errors (Pdr>Batcher>Log>pdr.log):
  1. Explore reasons for batcher errors.
  1. pdrBatcher inactivity timeout observed over a longer period?
  1. For example, after an AEMO bay transfer.
  2. Restart Batcher for threads to recover.
  1. FTP account validity:
  1. Update pdrBatcher.properties with a valid FTP account, host, password (correctly encrypted?) and restart the pdrBatcher.
  2. MarketNet access issues from the pdrBatcher host:
  • Try connecting to FTP directly and check authentication and response.
  1. Java runtime environment is correctly defined for the pdrBatcher:
  2. Update pdrEnvironment.bat (console mode) pdrServiceInstall.bat (running as service) for correct Java runtime path.
  3. Restart or reinstall the pdrBatcher service.
  1. Files are produced at the source:
  • FTP to AEMO and check the source directory for this file type.
  1. Local directory space issues:
  2. Make space available in the holding or reports directory?
  3. If multiple target directories are used ensure their existence and available space.
  4. Backup, clean up, archive log files to make space and restart the pdrBatcher.
  1. pdrBatcher Windows service or Unix daemon is running.
  2. pdrBatcher.properties file modified?
  • Restart the Batcher to make the changes effective.

pdrLoader is repeatedly re-requesting a file

The system is trying to load a file that was subscribed to but fails and and continues to re-request that file until it is loaded successfully. The recovery process is based on the subscriptions setup when the file was generated, not on the current time.

Before taking action to cease file requests, participants must determine the root cause and initiate remedial action required. A database space issue can result in this behaviour. Failure to properly investigate error conditions can result in data loss to the local database.

To stop re-requesting of the file, do either of the following:

  1. Correct the configuration error that is causing the file not to load. A successful file load will stop this file being re-requested.
  2. Use the pdrMonitor to remove the file from the request cycle as follows:
  • Menu Replication>Monitor Replication>Manifest tab. Right-click the problem file and set the status to ABANDONED.
  1. Alternatively, update the STATUS in PDR_MANIFEST_LOG to ABANDONED for the file, directly in the database.

When a file is in the ABANDONED state it is no longer requested, but the row is left in the PDR_MANIFEST_LOG table to record that there was an issue. You can manually delete this row from the database if you do not want to retain a permanent record.