Key aspects of observability for a Platform

Isolate Traces, Logs, Metrics so that we can focus on right problem

Traceability

1. Ability to trace origin of the calls - User / IP etc.
2. Request end to end traceability with unique identifier
3. Ability to measure duration of execution in each hop
4. One destination / system to capture and query for end to end visibility
5. Ability to persist data for agreed duration
6. One view to track health of your system by only showing the failures or slow responsive system

Logs

1. Isolate errors / exceptions
2. Ability to control debug / info / error levels
3. Avoid too much information
4. Understandable by anyone outside your team
5. Right categorization so that you can focus on finite data

Metrics

1. Ability to detect failures like HTTP 5xx etc.
2. Ability to detect anomalies in a given duration
3. Ability to be able to view and navigate through - Application and Infrastructure
4. If possible build a drill down
5. Don't forget - every level in the organization pyramid wants a different view of these metrics

One key aspect for validating completeness - ask support team or someone outside your team to validate the completeness of steps and documentation for triaging.