Evaluating LLMs’ Bayesian capabilities
As with people, to be efficient, an LLM’s person interactions require continuous updates to its probabilistic estimates of the person’s preferences based mostly on every new interplay with them. Right here we ask: do LLMs act as if they’ve probabilistic estimates which can be up to date as anticipated from optimum Bayesian inference? To the extent that the LLM’s conduct deviates from the optimum Bayesian technique, how can we reduce these deviations?
To check this, we used a simplified flight suggestion job, wherein the LLMs work together as assistants with a simulated person for 5 rounds. In every spherical, three flight choices have been offered to each the person and the assistant. Every flight was outlined by a departure time, a period, plenty of stops, and a price. Every simulated person was characterised by a set of preferences: for every characteristic, they may have a robust or weak choice for top or low values of the characteristic (e.g., they could favor longer or shorter flights), or no choice relating to this characteristic.
We in contrast the LLMs’ conduct to that of a mannequin, a Bayesian assistantthat follows the optimum Bayesian technique. This mannequin maintains a likelihood distribution that displays its estimates of the person’s preferences, and makes use of Bayes’ rule to replace this distribution as new details about the person’s selections turns into obtainable. Not like many real-life eventualities, the place it’s troublesome to specify and implement the Bayesian technique computationally, on this managed setting it’s simple to implement and permits us to exactly estimate the extent to which LLMs deviate from it.
The aim of the assistant was to suggest the flight that matches the person’s selection. On the finish of every spherical, the person indicated to the assistant whether or not or not it selected accurately, and offered it with the proper reply.
