Predicting Performance of Symbolic and Prompt Programs with Examples

This research paper introduces a coin-flip model to predict the performance of symbolic and prompt-based LLM programs using a few in-domain examples and a performance prior. It finds that symbolic programs exhibit an "all or nothing" performance prior, while prompt programs have a diffuse prior.