Logs, errors and codes
The engine logs one JSON line per event to standard output, through pino. Each line has an event
name and fields that are ids, counts, codes and error names: never a message's content, an error's own
text or a source file's name. Commands whose standard output is their report (status, messages,
rekey and the rest) send their log lines to standard error.
Log events
Messages
| Event | Level | Fields | Meaning |
|---|---|---|---|
received | info | channel, message, bytes, failed | a message is stored; failed when the source stored it as failed |
already-stored | info | channel, message | a source handed in again a message it had stored before a stop |
possible-duplicate | warn | channel, message, duplicateOf | same sender and control ID as an earlier message |
possible-duplicate | warn | channel, message, delivery, destination, reason, interruptedAttempt | a delivery was found cut off after its send (reason send-interrupted), and is sent again, flagged |
receive-failed | error | channel, code | a message could not be stored, and was not acknowledged |
transformed | info | channel, message, outputs | the transform returned outputs |
filtered | info | channel, message, outputs | the transform returned none |
transform-failed | warn | channel, message, kind, name, code, where | the message is marked failed; kind is the stage |
delivered | info | channel, destination, delivery, resend | a destination accepted an output; resend only on a flagged resend |
content-dropped | info | channel, message, delivery | the last delivery was delivered, and the message's content dropped (storage.content: dropAfterDelivery) |
dry-run | info | channel, destination, delivery | a destination that runs dry recorded an output and sent nothing |
delivery-retry | warn | channel, destination, delivery, attempt, retryInMs, code | it is tried again after retryInMs |
delivery-redirected | warn | channel, destination, delivery, attempt, retryInMs, status, locationHost | an HTTP destination answered a 3xx; held, never followed, and tried again after retryInMs (retry.max, 5 minutes by default) |
delivery-rejected | warn | channel, destination, delivery, reason | the destination rejected it for good |
delivery-exhausted | warn | channel, destination, delivery, attempt, reason, code | out of retries (reason limit or maxAge): marked failed, its body kept for a resend |
delivery-halted | warn | channel, destination, delivery, attempt | pathways channel halt cut the send off; it stays pending and is sent again, flagged, after a resume |
A transform-failed line's kind is load, parse, transform, output, timeout, memory or
crash. where is file:line:column in the transform, when it threw.
Sources and endpoints
| Event | Meaning |
|---|---|
source-start-failed | a source could not listen (code, such as EADDRINUSE); tried again after retryInMs |
source-started | a source that was down is listening again |
source-lost | a running source said it lost its connection (code, attempt); it should try again after retryInMs |
source-regained | a source that said it lost its connection has it again |
poll-deadline | a polled source's pass ran past its deadlineMs and was cut off; the source took nothing more in that pass |
poll-failed-after-cut | a polled source's poll call rejected after its pass was cut off, with an error of its own rather than the cut's (code); the cut goes on as before |
poll-overlap | a polled source's pass came due while its last poll call had not returned, and was skipped |
source-read-failed | a file source could not claim or read a file; it stays for the next pass |
source-file-set-aside | a file source will not take a file, by reason (too-large or unreadable); moved says whether it went into errorDir, and code why not |
http-refused | an HTTP source or the health endpoint refused a request, with its status |
http-unreadable | an HTTP source answering with ACKs stored a message as failed, with its problem |
mllp-listening | an MLLP source is listening, on port, with tls true or false |
mllp-store-failed | an MLLP message could not be stored; onStoreFailure says what was answered |
mllp-unreadable | an MLLP message was stored as failed, with its problem |
mllp-tls-refused | a TLS handshake on an MLLP source failed, such as a client without a certificate |
mllp-ack-builder-failed | a custom ACK builder threw or ran out of time; the default ACK was sent |
mllp-ack-unsendable | an ACK held an MLLP framing byte, so none was sent |
mllp-source-error, mllp-source-failed, mllp-security-warning | the MLLP server reported an error or a warning, by name and code |
Configuration and storage
| Event | Meaning |
|---|---|
plaintext | an endpoint runs without TLS (endpoint says which); logged at every start |
unauthenticated | a listening endpoint has neither a token nor tls.ca |
plaintext-copies | a file source keeps plaintext copies (keepProcessed: true) |
unencrypted-storage | encryption: none |
dry-run | pathways start --dry-run: every destination runs dry (destinations is all) |
dry-run-destinations | at start, the destinations that run dry, by index under each channel's name (channels); a destination not named sends |
dry-run-pending | a channel with a destination that runs dry (destinations, by index) has pending deliveries at start, of every destination; each of a dry destination is marked dry-run, not sent |
retention | a retention run, with the messages and deliveries it deleted |
retention-failed | a retention run failed, by code; the next run tries again |
lease-acquired | the engine took the engine lease, with its node and epoch |
lease-lost | the engine lost its lease, with the reason and epoch; it stops and exits 1 |
lease-unsupported | a custom store has no lease methods: the engine runs unfenced, and a second one is not refused |
lease-release-failed | the lease could not be freed at a stop, by code; it lapses instead |
standby | started as a standby, with its node |
standby-waiting | a standby is waiting for the lease another engine holds (holder, epoch) |
channel-state | a channel's state changed (running, paused or stopped), here or on another engine |
channel-states-unread | the channel states could not be read at the top of a pass, by code; the last ones stand |
channel-state-unstored | a pause, resume, stop or halt could not be stored, by code; the channel was put back as it was |
channel-reloaded | pathways channel reload swapped in the channel's new transform and settings |
Hooks and the audit log
| Event | Meaning |
|---|---|
hook-failed | a hook threw, with its name and code |
hook-timeout | a hook ran past its time limit |
hook-dropped | a hook had 1,000 events waiting, or was stopping, and events were dropped |
hook-stop-failed | a hook's stop() threw or ran out of time |
audit-listener-failed | a listener on the audit log failed; the entry stands |
Delivery codes
A retried delivery stores and logs a code. These come from Pathways' own connectors; a network error
keeps the operating system's code, such as ECONNREFUSED or ENOTFOUND.
| Code | Connector | Meaning |
|---|---|---|
CONNECT_TIMEOUT | MLLP | no connection within ackTimeout |
ACK_TIMEOUT | MLLP | no ACK within ackTimeout |
ACK_AR, ACK_CR | MLLP | the receiver answered with a reject code |
ACK_UNREADABLE | MLLP | the answer was not an ACK Pathways could read |
ACK_CONTROL_ID_MISMATCH | MLLP | the ACK answered another message's MSH-10 |
MLLP_CONNECTION | MLLP | the connection ended with no other code |
HTTP_<status> | HTTP | a 408, 425, 429 or 5xx answer, or a held 3xx |
ETIMEDOUT | HTTP | no answer within timeout |
DELIVERY_TIMEOUT | any | the attempt ran past its deadline, cut off |
A delivery-retry line's retryInMs is the backoff, or the wait a 429's or a 503's Retry-After
asked for when that is longer, up to the destination's retry.max, 5 minutes by default. A
delivery-exhausted line comes only from a destination whose retry settings set a limit or a
maxAge; see
Delivery and errors. A delivery-redirected line's locationHost is the
host (and port, when it names one) of the 3xx answer's Location, never its path or query, and is
absent when there is no Location or it names no host.
A delivery rejected for good stores its reason: the ACK code (AE, CE), HTTP <status> for a
101 or a 4xx other than 408, 425 and 429, or MLLP_PAYLOAD_CONTAINS_... for an output that
holds a framing byte.
MLLP problem codes
When an MLLP source stores a message as failed, the reason is one of @cosyte/hl7's parse codes (such
as NO_MSH_SEGMENT), NO_CONTROL_ID, MULTIPLE_MESSAGES or MLLP_TRAILING_BYTES. An ACK builder
receives it as problem.
File source codes
Besides the operating system's codes (EACCES, EEXIST and the like), a source-read-failed or
source-file-set-aside line from a file source can carry:
| Code | Meaning |
|---|---|
ERR_SCRIPT_EXECUTION_TIMEOUT | an include or exclude pattern ran past 50 ms on a path; the file is tried again on a later pass |
ESTALE | a file left the source folder, or was replaced, between the pass finding it and moving it; it stays |
ERR_PATHWAYS_FILE_HELD | a file moved in from outside the folder could not go back, and waits in .claimed/ as refused-... |
ERR_PATHWAYS_FILE_ID_UNSTEADY | the folder's file system changes a file's id when it moves (FAT, exFAT), so the source claims nothing there |
SFTP codes
A file source or destination on an SFTP server gives the SFTP status of a failed request as the
operating system's code where there is one (ENOENT, EACCES, ENOTSUP, and ELOOP for a link
where a file was opened), and otherwise one of these. The first nine, and a system code on
connecting (such as ECONNREFUSED), mark the source down until a pass runs again. A missing ssh2
package stops the engine at the first pass that needs it.
| Code | Meaning |
|---|---|
ERR_PATHWAYS_SFTP_HOST_KEY_UNKNOWN | the known hosts hold no key for the server; the error names its fingerprint |
ERR_PATHWAYS_SFTP_HOST_KEY_CHANGED | the server's key is not the one the known hosts or hostKey hold |
ERR_PATHWAYS_SFTP_HOST_KEY_REVOKED | the server's key is on an @revoked line |
ERR_PATHWAYS_SFTP_AUTH | the server refused the login |
ERR_PATHWAYS_SFTP_KEY_INVALID | the private key could not be read, or its passphrase is wrong |
ERR_PATHWAYS_SFTP_NO_POSIX_RENAME | the server has no posix-rename@openssh.com, so a claim cannot be made |
ERR_PATHWAYS_SFTP_TIMEOUT | the connection, or one request, took longer than its deadline |
ERR_PATHWAYS_SFTP_CONNECTION_LOST | the connection dropped while it was in use |
ERR_PATHWAYS_SFTP_CONNECT | the connection could not be made, with no system code |
ERR_PATHWAYS_SFTP_NOT_INSTALLED | the ssh2 package is not installed |
ERR_PATHWAYS_SFTP_SIZE_MISMATCH | a read or a write moved a different number of bytes than the file's size |
ERR_PATHWAYS_SFTP_FAILURE | the server failed a request without saying why |
ERR_PATHWAYS_SFTP_EOF | the server said the file ended where it should not have |
ERR_PATHWAYS_SFTP_BAD_MESSAGE | the server could not read a request |
ERR_PATHWAYS_SFTP_STATUS_<n> | any other SFTP status, by number |
Encryption errors
| Code | Meaning |
|---|---|
ERR_PATHWAYS_NOT_ENCRYPTED | a stored value is not encrypted: it was stored while encryption was none, or changed |
ERR_PATHWAYS_ENCRYPTED | a stored value is encrypted, and encryption is none |
ERR_PATHWAYS_MALFORMED | a stored value starts like an encrypted one, but its layout is broken |
ERR_PATHWAYS_FORMAT_VERSION | a stored value is in an encrypted format this version does not read |
ERR_PATHWAYS_UNKNOWN_KEY | the master key a value names is not configured |
ERR_PATHWAYS_INTEGRITY | a value, or its wrapped data key, failed its integrity check: changed, damaged, or the key under its id is not the one that wrote it |
ERR_PATHWAYS_KEY_PROVIDER | a key provider returned a key id or a key that cannot be used |
Related
- Troubleshooting.
- Command reference: exit codes.
- Reference:
EncryptionErrorCode.