idbd

IDBD · Sutton, 1992

A learner that works out what to listen to.

IDBD is a rule for learning from a stream of data. It keeps a separate learning speed for every input and tunes those speeds as it goes, so it reacts fast to what matters and tunes out what doesn't.

Show me how Play against it
Advanced mode: the full Explorer → The original paper: Sutton, 1992, AAAI-92 (PDF) ↗
GitHub minikin/idbd crates.io idbd
A cutaway home with twelve sensors. Each sensor's amber ring is as wide as the learning speed IDBD gives it.
    Amber ring: how closely IDBD watches a sensor Bigger ring, faster learning per typical swing of a reading. Nobody told it which sensors matter: it watched this home for 7 days one reading a minute before you arrived.
    Power meter above the base, liveday 1, 00:00

    actual IDBD's guess

    What IDBD does

    Guess and check. Decide how hard to correct, then correct.

    A home assistant wants to predict the power meter. Every minute it reads its twelve sensors, guesses the power draw, sees what the meter really says and corrects itself. Most learning programs work this way. IDBD adds one thing: every sensor gets its own learning speed, and IDBD tunes those speeds before every correction. Below, the four steps run on the same home, one reading at a time.

    1. 1

      Guess

      Multiply each reading by the trust in its sensor and add them up.

    2. 2

      Check

      Read the meter. The difference from the guess is the miss.

    3. 3

      Tune the speeds

      Compare each sensor's correction with its recent ones. If they keep pushing the same way, speed that sensor up. If they keep cancelling out, slow it down. This step is IDBD.

    4. 4

      Correct

      Move each trust to shrink the miss, as far as that sensor's speed allows, and remember the move so the next reading can be compared with it.

    01 · Too many signals

    Two thirds of what this home measures never touches the bill.

    The assistant reads twelve sensors every minute: thermometers, the Wi-Fi router, the doorbell, the front door, the air, even the plant's soil. Four of them add to the power bill, each during its own hours of the day. The other eight never do.

    Nobody tells the learner which four, or when. It has to work that out from the readings alone. And the sensors read on different scales: some swing four times wider than others.

    12 sensors4 matter0 labels

      Every sensor wiggles. Which four cost you money, and when? You get to guess in chapter 4.

      live readings from the simulated home

      02 · One speed is a bad compromise

      A single learning speed is either late or nervous.

      A plain learner corrects every sensor with one fixed step size. In this home the sensors read on different scales, so one step size is slow on sensors with small readings and jumpy on sensors with large ones. Here are two of them, unnamed so the game stays a game.

      The tuned fixed speed, 0.01calm, but late
      before↓ the sensor starts to matter↓ stops
      what the sensor really addsfixed-speed trustIDBD's trust
      A sensor with small readings.
      A faster fixed speed, 0.05quick, but nervous
      beforesame minutes
      what the sensor really adds: nothingfixed-speed trustIDBD's trust
      A sensor with large readings that never matters.

      What we want: quick on the small readings, still on the large ones. One step size cannot do both. IDBD keeps one per sensor.

      measured by the idbd crate: home seed 1, day 2

      03 · Slow motion

      Watch IDBD learn, one reading at a time.

      This learner starts knowing nothing, with the same step size for every sensor. Press Next to give it one reading. Press Run three days to skip ahead through all 4,320 readings and watch the speeds spread apart.

      Every number is per typical swing of that sensor's reading, so the sensors compare directly: a reading of +1.00 is one typical swing above its usual value, and the guess is each reading times its trust, added up.

      Sensor names stay hidden until you have played the game in chapter 4.

      Reading0
      IDBD guessed the meter would read-
      The meter really read-
      The guess was-

      Press Next reading to start.

      Scroll the table sideways for the speeds →

      Sensor Reading, in typical swings Trust, kW per swing Speed per typical swing Speed so far

      04 · Play

      Family Assistant: which sensors cost you, and when?

      A home with twelve sensors and one power meter. Four of the sensors add to the bill, each during its own hours of the day. The other eight never do. Mark the hours you think each sensor matters, then let IDBD watch the home for three days and see what it found on its own.

      Drag the two handles on a lane to mark its hours. Leave a lane empty if you think the sensor never matters.

        05 · What IDBD learned

        After three days, IDBD listens closely to four sensors and lets eight go quiet.

        Nobody told it which sensors matter. The corrections for the water heater, the wall, the oven and the Wi-Fi data use kept agreeing, and the others kept cancelling out.

        Speeds rise a little in the hours a sensor is busy, but not enough to tell those hours apart. The game's hour bars come from the trust, not the speed.

        Richard Sutton called the rule Incremental Delta-Bar-Delta in 1992.

        Speed per typical swing IDBD learned for each sensor after 4,320 readings, the game's first run simulated home, computed by the idbd crate

          Advanced mode

          The full Explorer.

          Step through one IDBD update term by term, run IDBD, LMS and Autostep side by side on Sutton's tasks, change the world, replay and export runs.

          Open the Explorer