Advanced~3h

Observability

Put a number on the telemetry

Every course on this subject states the arithmetic and never asks for it. Here you do it: how many time series a label set creates and which label explodes it, how many failures an objective allows and what an incident spent, and which of four alerts can fire while nothing is wrong.

Start this project See the modules

Modules: 4
Drills: 5
Build steps: 0
Time: ~3h

Walk out able to

Price a proposed label in series, cost an incident against a stated objective, and say which alerts on a page can lie — in both directions.

What you work on

Logs, metrics and traces

Which one can answer this, as it was actually recorded.

Cardinality, multiplied out

What a label set costs in series, before it ships.

Alerts against a promise

Which fire for nothing, and which sleep through a miss.

Error budgets, divided out

What the objective allows, and what Tuesday spent.

The pitch

What you practise, and what you leave with

An objective sheet for a service you actually run, and a costed incident: series added, budget spent, and the one rule worth keeping.

You will practise

Route a question to the signal that can answer it
Multiply a label set out to time series
Find the alert that fires while nothing is wrong
Turn an objective into seconds, and spend them

Afterwards you can

Price a new metric label before it ships
Say what a sampled signal can never count
Compute when a windowed alert first pages
Cost an incident against its error budget

Modules

4 modules, 20 items

Lessons explain one idea. Drills repeat it until it sticks. Build steps make something that exists afterwards.

1
Which signal can answer this
~45 min
Hold the instrumentation fixed and say what it can and cannot answer.
- LessonWhy put a number on telemetry5 min
- LessonAsk what the telemetry can answer9 min
- LessonWhat sampling can never count7 min
- LessonFind the time a waterfall hides8 min
- LessonChildren that overlap do not add7 min
- DrillCount the traces one request leaves9 min
2
What a dimension costs
~35 min
Multiply a label set out to time series, and name the label carrying the cost.
- LessonMultiply a label set out to series10 min
- LessonWhere an identifier belongs7 min
- DrillCount a histogram’s series8 min
- DrillName the label carrying the cost10 min
3
Alerts that mean something
~41 min
Test a rule against the objective: can it fire for nothing, or sleep through a miss?
- LessonFind the two ways an alert lies9 min
- LessonWhat a page is actually for7 min
- DrillTime the page from the window10 min
- LessonTurn a failure ratio into a deadline8 min
- LessonWhat a fast-burn rule cannot see7 min
4
Spend the budget
~59 min
Turn an objective into an allowance, cost an incident against it, write it down.
- LessonTurn an objective into an allowance8 min
- LessonPrice one more nine7 min
- DrillCost an incident against the budget10 min
- LessonWrite the sheet for a real service16 min
- LessonCost one incident end to end18 min