This article explores the relationship between entropy, compression, and intelligence, serving as the first part of a series. It aims to redefine the understanding of these fundamental concepts.
This paper introduces PromptNCE, a method to estimate pointwise mutual information (PMI) using only LLMs and contrastive estimation prompts, circumventing the need for task-specific critics. It presents a benchmark with human-derived PMI and shows PromptNCE achieves Spearman correlation up to 0.82.
This paper explores detection within the stochastic block model with multiple clusters, providing proofs for achievability conjectures. It also discusses acyclic Belief Propagation and information-theoretic aspects of the model.
This article, a re-publication of Rei-AIOS Paper 110, presents a rigorous information-theoretic comparison between the Braille-Unicode × D-FUMT8 encoding and multimodal embedding schemes like CLIP, BERT, and ImageBind. The research explores representing 256 philosophical states within a single 3-byte UTF-8 character.
Este artigo investiga a correlação entre a dinâmica interna de entropia e o raciocínio correto em Large Language Models (LLMs), um enigma ainda sem solução. Propõe a Hipótese de Informatividade Gradual (SIA), que afirma que os modelos raciocinam corretamente ao acumular informações relevantes sobre a resposta por meio de prefixos informativos, um processo reforçado por métodos de treinamento padrão.
This research paper posits the origin of synthetic information as a core mystery in information science, drawing an analogy to the origin of species. It introduces a steganographic inheritance mechanism to help trace the evolutionary lineage of AI-generated synthetic information, acknowledging the moral implications and technical challenges.
This paper introduces bi-predictability (P) and the Information Digital Twin (IDT) architecture for real-time monitoring of LLM interaction integrity. It aims to continuously ensure structural coupling in multi-turn workflows, addressing the shortcomings of current evaluation methods that fail to detect gradual degradation.
This article reframes the Fujimoto Infinite Dot Theory (FIDT) from a universal codec to a domain-specific generator for D-FUMT₈ theories. This new positioning, developed with Claude Opus 4.7's collaboration, achieves byte-exact reconstruction and high compression.
The paper proposes a neural framework to estimate pairwise conditional mutual information (MI) directly from the hidden states of pretrained masked diffusion models (MDMs). This method captures dependency structures and enables MI-guided parallel decoding, showing utility in Sudoku and protein sequence generation by recovering known structural constraints.
This paper introduces predictable history-adaptive virtual perturbations to enhance information-theoretic generalization bounds for Stochastic Gradient Descent. This new approach allows perturbation covariances to dynamically depend on past SGD history, addressing limitations of existing methods that require fixed covariances.
This content explores notes on Kullback-Leibler Divergence and its relationship with the concept of Likelihood. It covers fundamental principles of information theory and statistical inference relevant to AI.
This paper formulates masking-based AI explanation methods as communication over a query channel, where explanations act as messages. It derives information-theoretic limits for the recovery of exact explanations, showing that reliable recovery is achievable below a certain capacity.
This content provides an intuitive understanding of Entropy and its applications in Machine Learning and Data Analysis. It also includes Python code examples to facilitate learning.
This content provides an introduction to Information Theory, focusing on its applications in Machine Learning and Data Analysis. Python code is included to aid understanding.
The text discusses the long history of indirect and distorted communication between the US and Iran through intermediaries. Current negotiations in Islamabad represent a pioneering attempt at direct dialogue to correct decades of 'translation loss'.