Skip to content

Troubleshooting Policy Engine Installation

Installation Troubleshooting

Functional Troubleshooting

ACM Success logs

  • Check the logs in ARS for below scenarios

  • Certificate not found and generating new

    • No valid certificate data for user '<Username>' was found in the database; generating a new certificate.

    • A new user certificate for '<Username>' (serial no. <serial>) has been generated by Vault.

  • Reusing certificate

    • An existing user certificate for '<Username>' (serial no. <serial>) was found in the database; it will be reused.

    • An existing user certificate for' <Username>' (serial no. <serial>) was retrieved from Vault.

ACM Error logs

Application Startup

Log Message: "Failed to initialize database"
  • Scenario: The application fails to connect to the database during startup.
  • Possible Reasons:
    • Database server is down or unreachable
    • Database configuration (host, port, credentials) is incorrect
    • Network connectivity issues between application and database
    • Database authentication credentials are invalid or expired
    • Database service is not running or misconfigured
  • Impact : The application cannot start; no data operations are possible; complete application startup failure.
Log Message: "Failed to initialize CA connector"
  • Scenario: The application cannot connect to the Certificate Authority (CA) system (e.g., Vault) on startup.
  • Possible Reasons:
    • CA service (Vault) is down or unreachable
    • CA connector credentials are missing or incorrect
    • Network connectivity issues to CA service
    • TLS/SSL certificate issues for secure connection
  • Impact : The application cannot manage certificates; startup fails; no certificate operations can be performed.
Log Message: "Invalid CA connector"
  • Scenario: The CA connector is not properly set up during initialization.
  • Possible Reasons:
    • CA connector was not initialized during startup
    • CA connector initialization failed silently
    • Configuration error preventing CA connector setup
    • Dependency injection or service creation failure
  • Impact : Certificate management services are unavailable; application will fail to start or operate properly.
Log Message: "Vault instance is unhealthy" (Warning)
  • Scenario: A health check indicates that a Vault instance is not responding properly.
  • Possible Reasons:
    • Vault service is down or restarting
    • Vault is sealed and needs to be unsealed
    • Network connectivity issues to Vault
    • Vault is overloaded or experiencing performance issues
    • Vault configuration or permission problems
  • Impact : Vault instance is marked as unavailable; requests are routed to other healthy instances; certificate operations may be affected if no healthy instances are available.

Token Validation

Log Message: "Token is invalid or expired"
  • Scenario: The authentication token provided in the request is not valid.
  • Possible Reasons: i: Token has expired and needs renewal ii: Token signature is invalid or corrupted iii: Token was issued by different authority or with wrong keys iv: Token format is incorrect or malformed v: Token was revoked or blacklisted
  • Impact : User authentication fails; request is rejected with unauthorized error; user must re-authenticate.
Log Message: "Tenant keys not found in database"
  • Scenario: System cannot find encryption keys for specific tenant.
  • Possible Reasons:
    • Tenant keys never created
    • Keys deleted or corrupted
    • Wrong tenant ID provided
    • Database query failed
  • Impact : Certificate encryption/decryption fails, authentication fails no cert is generated

Request Validation

Log Message: "Invalid request body"
  • Scenario: The request body cannot be read, parsed, or decrypted properly.
  • Possible Reasons: i: Request body is empty when data is expected ii: JSON format is invalid or malformed iii: Encryption/decryption of request body failed iv: Character encoding issues or corrupted data
  • Impact : Request cannot be processed; user receives bad request error; operation fails.
Log Message: "Missing X-Correlation-Id header, generated new ID" (Warning)
  • Scenario: Request lacks correlation ID for tracking.
  • Possible Reasons:
    • Client didn't include header
    • ID lost during routing
    • Client unaware of requirement
    • API gateway stripped header
  • Impact : New correlation ID generated; minimal functional impact.
Log Message: "Validation failed for request"
  • Scenario: The request data doesn't meet the required validation criteria.
  • Possible Reasons: i: Required fields are missing from the request ii: Business logic validation rules are violated iii: Data type mismatches or format errors
  • Impact : Request is rejected; user must correct input data; operation cannot proceed until valid data is provided.
Log Message: "Invalid request context" (very rare)
  • Scenario: The request context exists but is in an unexpected format or type.
  • Possible Reasons:
    • Middleware set context with wrong data type
  • Impact : Request processing fails; user request is rejected; system cannot identify tenant or customer information.

Certificate Generation

Log Message: "Invalid request parameters"
  • Scenario: The certificate generation request contains invalid or missing parameters.
  • Possible Reasons:
    • Client sent request with empty or null userId
    • PartitionKey (role) is missing or empty in the request
    • Request parsing failed, and parameters were lost
    • Client-side validation bypassed or failed
    • Parameter values are outside acceptable ranges
  • Impact : Certificate generation request is immediately rejected with a bad request error; no certificate processing occurs.
Log Message: "Certificate not found in database"
  • Scenario: The system tries to retrieve a certificate from the database but it doesn't exist.
  • Possible Reasons:
    • Certificate was never created or stored
    • Certificate was deleted or expired
    • Wrong certificate ID provided in request
    • Database query failed or returned no results
  • Impact : Certificate operation fails; user cannot access, revoke, or manage the requested certificate.
Log Message: "Failed to process existing certificate, attempting to generate new one" (Warning)
  • Scenario: An existing certificate is found but cannot be processed.
  • Possible Reasons:
    • Private key decryption failed
    • Certificate retrieval from Vault failed
    • Certificate data is corrupted
    • PFX generation failed
    • Certificate has expired or is invalid
  • Impact : Existing certificate is discarded; system attempts to generate new certificate; may cause temporary delay.
Log Message: "Failed to delete invalid certificate from database" (Warning)
  • Scenario: System fails to remove invalid certificate from database.
  • Possible Reasons:
    • Database connection was dicsonnected
    • databse is down because of some issue
  • Impact : Invalid certificate remains in database;
Log Message: "Certificate generation failed"
  • Scenario: System fails to generate new certificate through Vault.
  • Possible Reasons:
    • Vault service rejected the request
    • Invalid certificate parameters
    • Vault service is down
    • Insufficient Vault permissions
    • Network timeout with Vault
  • Impact : Certificate generation completely fails; user cannot obtain certificate.
Log Message: "Failed to generate certificate after all attempts"
  • Scenario: All certificate generation attempts failed.
  • Possible Reasons:
    • All vault resouces down
    • database resources down
  • Impact : Certificate generation completely fails

PFX Generation

Log Message: "Failed to decode certificate PEM"
  • Scenario: System cannot read certificate in PEM format.
  • Possible Reasons:
    • Certificate data corrupted
    • Invalid PEM format
  • Impact : PFX generation fails; certificate cannot be packaged.
Log Message: "Failed to parse certificate"
  • Scenario: System cannot interpret certificate data structure.
  • Possible Reasons:
    • Non-standard X.509 format
    • Data corrupted
    • Unsupported version/extensions
  • Impact : Certificate validation and PFX generation fail.
Log Message: "Failed to decode private key PEM"
  • Scenario: System cannot read private key in PEM format.
  • Possible Reasons:
    • Private key data corrupted
    • Invalid PEM format
    • Unsupported algorithm
    • Encoding issues
  • Impact : Private key cannot be processed; PFX generation fails.
Log Message: "Failed to parse private key"
  • Scenario: System cannot interpret private key data structure.
  • Possible Reasons:
    • Unsupported format
    • Key data corrupted
    • Wrong algorithm/parameters
  • Impact : Private key operations fail; PFX generation cannot proceed,cert gen fails
Log Message: "Failed to decode issuing CA certificate PEM"
  • Scenario: System cannot read CA certificate in PEM format.

  • Possible Reasons:

    • CA certificate data corrupted
    • Invalid PEM format

  • Impact : Certificate chain cannot be built; PFX bundle generation fails,cert gen fails

Log Message: "Failed to parse issuing CA certificate"
  • Scenario: System cannot interpret CA certificate data structure.
  • Possible Reasons:
    • Invalid format
    • Data corruption
  • Impact : Certificate chain validation fails; PFX bundle creation fails,cert gen fails
Log Message: "Failed to decode intermediate CA certificate PEM"
  • Scenario: System cannot read intermediate CA certificate in PEM format.
  • Possible Reasons:
    • Intermediate certificate corrupted
    • Invalid PEM format
  • Impact : Complete certificate chain cannot be built.
Log Message: "Failed to parse intermediate CA certificate"
  • Scenario: System cannot interpret intermediate CA certificate structure.
  • Possible Reasons:
    • Invalid format
    • Data corruption
  • Impact : Certificate chain is incomplete; PFX generation fails,cert gen fails
Log Message: "Failed to generate PFX file"
  • Scenario: System cannot create PFX certificate bundle.
  • Possible Reasons:
    • Invalid certificate/key data
    • Missing/invalid PFX password
    • Certificate/key mismatch
  • Impact : Certificate bundle cannot be created; entire operation fails,cert gen fails

Certificate Cleanup Errors

Log Message: "Failed to schedule cleanup job"
  • Scenario: Cron job scheduling fails during service initialization.
  • Possible Reasons:
    • Invalid cron expression format in configuration
    • Cron scheduler initialization failure
  • Impact: Automatic certificate cleanup will not occur; manual intervention required; expired certificates accumulate.
Log Message: "Failed to fetch certificates"
  • Scenario: Database query to retrieve certificates fails during cleanup process.
  • Possible Reasons:
    • MongoDB connection lost during query execution
    • Database query timeout (20-second timeout exceeded)
    • Invalid query parameters (customerId, tenantId)
    • Insufficient database permissions for read operations
  • Impact: Cleanup process halts; expired certificates not identified; cleanup incomplete; process retries on next scheduled run.
Log Message: "Failed to delete certificates from MongoDB"
  • Scenario: Batch deletion of certificates from MongoDB fails.
  • Possible Reasons:
    • MongoDB connection interrupted during delete operation
    • Insufficient database permissions for delete operations
    • Invalid serial numbers in deletion batch
  • Impact: Expired certificates remain in database; storage space not freed; data inconsistency possible; cleanup marked as failed.
Log Message: "Failed to initiate Vault PKI tidy"
  • Scenario: Vault PKI tidy operation cannot be initiated for certificate cleanup.
  • Possible Reasons:
    • Vault server is unreachable or experiencing connectivity issues
    • Vault authentication token expired or invalid
    • Insufficient Vault permissions for PKI tidy operations
    • Vault instance is sealed
    • PKI backend not properly configured or mounted
    • Network timeout during Vault API call
  • Impact: Vault certificate storage not cleaned; revoked certificates remain in Vault; storage bloat continues; cleanup partially successful.