Troubleshooting Policy Engine Installation

Installation Troubleshooting

Functional Troubleshooting

ACM Success logs

• Check the logs in ARS for below scenarios.

• Certificate not found, and generating a new one.

  • No valid certificate data for user '<Username>' was found in the database; generating a new certificate.

  • A new user certificate for '<Username>' (serial no. <serial>) has been generated by Vault.

• Reusing certificate

  • An existing user certificate for '<Username>' (serial no. <serial>) was found in the database; it will be reused.

  • An existing user certificate for' <Username>' (serial no. <serial>) was retrieved from Vault.

ACM Error logs

Application Startup

Log Message: "Failed to initialize database"

• Scenario: The application fails to connect to the database during startup.
• Possible Reasons:
  • The database server is down or unreachable.
  • Database configuration (host, port, credentials) is incorrect.
  • Network connectivity issues between the application and the database.
  • Database authentication credentials are invalid or expired.
  • The database service is not running or is misconfigured.
• Impact: The application cannot start; no data operations are possible; complete application startup failure.

Log Message: "Failed to initialize CA connector"

• Scenario: The application cannot connect to the Certificate Authority (CA) system (e.g., Vault) on startup.
• Possible Reasons:
  • CA service (Vault) is down or unreachable.
  • CA connector credentials are missing or incorrect.
  • Network connectivity issues with the CA service.
  • TLS/SSL certificate issues for a secure connection.
• Impact: The application cannot manage certificates; startup fails; no certificate operations can be performed.

Log Message: "Invalid CA connector"

• Scenario: The CA connector is not properly set up during initialization.
• Possible Reasons:
  • The CA connector was not initialized during startup.
  • CA connector initialization failed silently.
  • Configuration error preventing CA connector setup.
  • Dependency injection or service creation failure.
• Impact: Certificate management services are unavailable; the application will fail to start or operate properly.

Log Message: "Vault instance is unhealthy" (Warning)

• Scenario: A health check indicates that a Vault instance is not responding properly.
• Possible Reasons:
  • Vault service is down or restarting.
  • Vault is sealed and needs to be unsealed.
  • Network connectivity issues with Vault.
  • Vault is overloaded or experiencing performance issues.
  • Vault configuration or permission problems.
• Impact: Vault instance is marked as unavailable; requests are routed to other healthy instances; certificate operations may be affected if no healthy instances are available.

Token Validation

Log Message: "Token is invalid or expired"

• Scenario: The authentication token provided in the request is not valid.
• Possible Reasons: i: Token has expired and needs renewal. ii: Token signature is invalid or corrupted. iii: Token was issued by a different authority or with the wrong keys. iv: Token format is incorrect or malformed. v: Token was revoked or blacklisted.
• Impact: User authentication fails; request is rejected with unauthorized error; user must re-authenticate.

Log Message: "Tenant keys not found in database"

• Scenario: System cannot find encryption keys for a specific tenant.
• Possible Reasons:
  • Tenant keys were never created.
  • Keys deleted or corrupted.
  • Wrong tenant ID provided.
  • Database query failed.
• Impact: Certificate encryption/decryption fails, authentication fails, and no cert is generated.

Request Validation

Log Message: "Invalid request body"

• Scenario: The request body cannot be read, parsed, or decrypted properly.
• Possible Reasons: i: Request body is empty when data is expected. ii: JSON format is invalid or malformed. iii: Encryption/decryption of request body failed. iv: Character encoding issues or corrupted data.
• Impact: Request cannot be processed; user receives bad request error; operation fails.

Log Message: "Missing X-Correlation-Id header, generated new ID" (Warning)

• Scenario: Request lacks a correlation ID for tracking.
• Possible Reasons:
  • The client didn't include a header.
  • ID lost during routing.
  • The client is unaware of the requirement.
  • API gateway stripped header.
• Impact: New correlation ID generated; minimal functional impact.

Log Message: "Validation failed for request"

• Scenario: The request data doesn't meet the required validation criteria.
• Possible Reasons: i: Required fields are missing from the request. ii: Business logic validation rules are violated. iii: Data type mismatches or format errors.
• Impact: Request is rejected; user must correct input data; operation cannot proceed until valid data is provided.

Log Message: "Invalid request context" (very rare)

• Scenario: The request context exists but is in an unexpected format or type.
• Possible Reasons:
  • The middleware set the context with the wrong data type.
• Impact: Request processing fails; user request is rejected; system cannot identify tenant or customer information.

Certificate Generation

Log Message: "Invalid request parameters"

• Scenario: The certificate generation request contains invalid or missing parameters.
• Possible Reasons:
  • The client sent a request with an empty or null userID.
  • The Partition Key (role) is missing or empty in the request.
  • Request parsing failed, and parameters were lost.
  • Client-side validation bypassed or failed.
  • Parameter values are outside acceptable ranges.
• Impact: Certificate generation request is immediately rejected with a bad request error; no certificate processing occurs.

Log Message: "Certificate not found in database"

• Scenario: The system tries to retrieve a certificate from the database, but it doesn't exist.
• Possible Reasons:
  • The certificate was never created or stored.
  • The certificate was deleted or expired.
  • Wrong certificate ID provided in the request.
  • Database query failed or returned no results.
• Impact: Certificate operation fails; user cannot access, revoke, or manage the requested certificate.

Log Message: "Failed to process existing certificate, attempting to generate new one" (Warning)

• Scenario: An existing certificate is found, but cannot be processed.
• Possible Reasons:
  • Private key decryption failed.
  • Certificate retrieval from Vault failed.
  • Certificate data is corrupted.
  • PFX generation failed.
  • The certificate has expired or is invalid.
• Impact: Existing certificate is discarded; system attempts to generate a new certificate; may cause a temporary delay.

Log Message: "Failed to delete invalid certificate from database" (Warning)

• Scenario: System fails to remove an invalid certificate from the database.
• Possible Reasons:
  • The database connection was disconnected.
  • The database is down because of an issue.
• Impact: Invalid certificate remains in the database.

Log Message: "Certificate generation failed"

• Scenario: System fails to generate a new certificate through Vault.
• Possible Reasons:
  • Vault service rejected the request.
  • Invalid certificate parameters.
  • Vault service is down.
  • Insufficient Vault permissions.
  • Network timeout with Vault.
• Impact: Certificate generation completely fails; the user cannot obtain a certificate.

Log Message: "Failed to generate certificate after all attempts"

• Scenario: All certificate generation attempts failed.
• Possible Reasons:
  • All vault resources are down.
  • Database resources are down.
• Impact: Certificate generation completely fails.

PFX Generation

Log Message: "Failed to decode certificate PEM"

• Scenario: System cannot read the certificate in PEM format.
• Possible Reasons:
  • Corrupted certificate data.
  • Invalid PEM format.
• Impact: PFX generation fails; certificate cannot be packaged.

Log Message: "Failed to parse certificate"

• Scenario: The System cannot interpret the certificate data structure.
• Possible Reasons:
  • Non-standard X.509 format.
  • Data corrupted.
  • Unsupported version/extensions.
• Impact: Certificate validation and PFX generation fail.

Log Message: "Failed to decode private key PEM"

• Scenario: System cannot read the private key in PEM format.
• Possible Reasons:
  • Private key data corrupted.
  • Invalid PEM format.
  • Unsupported algorithm.
  • Encoding issues.
• Impact: Private key cannot be processed; PFX generation fails.

Log Message: "Failed to parse private key"

• Scenario: System cannot interpret the private key data structure.
• Possible Reasons:
  • Unsupported format.
  • Key data corrupted.
  • Wrong algorithm/parameters.
• Impact: Private key operations fail; PFX generation cannot proceed, cert gen fails.

Log Message: "Failed to decode issuing CA certificate PEM"

• Scenario: System cannot read the CA certificate in PEM format.

• Possible Reasons:

  • CA certificate data corrupted.
  • Invalid PEM format.

• Impact: Certificate chain cannot be built; PFX bundle generation fails, cert gen fails.

Log Message: "Failed to parse issuing CA certificate"

• Scenario: System cannot interpret the CA certificate data structure.
• Possible Reasons:
  • Invalid format.
  • Data corruption.
• Impact: Certificate chain validation fails; PFX bundle creation fails, cert gen fails.

Log Message: "Failed to decode intermediate CA certificate PEM"

• Scenario: System cannot read the intermediate CA certificate in PEM format.
• Possible Reasons:
  • The intermediate certificate is corrupted.
  • Invalid PEM format.
• Impact: A complete certificate chain cannot be built.

Log Message: "Failed to parse intermediate CA certificate"

• Scenario: System cannot interpret the intermediate CA certificate structure.
• Possible Reasons:
  • Invalid format.
  • Data corruption.
• Impact: Certificate chain is incomplete; PFX generation fails, cert gen fails.

Log Message: "Failed to generate PFX file"

• Scenario: System cannot create PFX certificate bundle.
• Possible Reasons:
  • Invalid certificate/key data.
  • Missing/invalid PFX password.
  • Certificate/key mismatch.
• Impact: Certificate bundle cannot be created; entire operation fails, cert gen fails.

Certificate Cleanup Errors

Log Message: "Failed to schedule cleanup job"

• Scenario: Cron job scheduling fails during service initialization.
• Possible Reasons:
  • Invalid cron expression format in configuration.
  • Cron scheduler initialization failure.
• Impact: Automatic certificate cleanup will not occur; manual intervention is required; expired certificates accumulate.

Log Message: "Failed to fetch certificates"

• Scenario: Database query to retrieve certificates fails during the cleanup process.
• Possible Reasons:
  • MongoDB connection lost during query execution.
  • Database query timeout (20-second timeout exceeded).
  • Invalid query parameters (customerID, tenantID).
  • Insufficient database permissions for read operations.
• Impact: Cleanup process halts; expired certificates not identified; cleanup incomplete; process retries on next scheduled run.

Log Message: "Failed to delete certificates from MongoDB"

• Scenario: Batch deletion of certificates from MongoDB fails.
• Possible Reasons:
  • MongoDB connection interrupted during the delete operation.
  • Insufficient database permissions for delete operations.
  • Invalid serial numbers in the deletion batch.
• Impact: Expired certificates remain in the database; storage space is not freed; data inconsistency is possible; cleanup is marked as failed.

Log Message: "Failed to initiate Vault PKI tidy"

• Scenario: Vault PKI tidy operation cannot be initiated for certificate cleanup.
• Possible Reasons:
  • Vault server is unreachable or experiencing connectivity issues.
  • Vault authentication token expired or is invalid.
  • Insufficient Vault permissions for PKI tidy operations.
  • Vault instance is sealed.
  • PKI backend not properly configured or mounted.
  • Network timeout during Vault API call.
• Impact: Vault certificate storage not cleaned; revoked certificates remain in Vault; storage bloat continues; cleanup partially successful.