A Decomposition Method for Global Evaluation of Shannon Entropy and Local Estimations of Algorithmic Complexity

dc.contributor.authorZenil, Hector
dc.contributor.authorHernández-Orozco, Santiago
dc.contributor.authorKiani, Narsis
dc.contributor.authorSoler-Toscano, Fernando
dc.contributor.authorRueda-Toicen, Antonio
dc.contributor.authorTegner, Jesper
dc.contributor.departmentBiological and Environmental Sciences and Engineering (BESE) Division
dc.contributor.departmentBioscience Program
dc.contributor.departmentComputer, Electrical and Mathematical Sciences and Engineering (CEMSE) Division
dc.contributor.institutionDepartment of Computer Science, University of Oxford, Oxford, OX1 3QD, , United Kingdom
dc.contributor.institutionAlgorithmic Nature Group, Laboratoire de Recherche Scientifique (LABORES) for the Natural and Digital Sciences, Paris, 75005, , France
dc.contributor.institutionAlgorithmic Dynamics Lab, Unit of Computational Medicine, Department of Medicine Solna, Center for Molecular Medicine, Karolinska Institute and SciLifeLab, Stockholm, SE-171 77, , Sweden
dc.contributor.institutionPosgrado en Ciencia e Ingeniería de la Computación, Universidad Nacional Autónoma de México (UNAM), Mexico City, 04510, , Mexico
dc.contributor.institutionGrupo de Lógica, Lenguaje e Información, Universidad de Sevilla, Seville, 41004, , Spain
dc.contributor.institutionInstituto Nacional de Bioingeniería, Universidad Central de Venezuela, Caracas, 1051, , Venezuela
dc.contributor.institutionUnit of Computational Medicine, Department of Medicine Solna, Center for Molecular Medicine, SciLifeLab and Karolinska Institute, Stockholm, SE-171 77, , Sweden
dc.date.accessioned2018-09-26T13:28:32Z
dc.date.available2018-09-26T13:28:32Z
dc.date.issued2018-08-15
dc.description.abstractWe investigate the properties of a Block Decomposition Method (BDM), which extends the power of a Coding Theorem Method (CTM) that approximates local estimations of algorithmic complexity based on Solomonoff-Levin's theory of algorithmic probability providing a closer connection to algorithmic complexity than previous attempts based on statistical regularities such as popular lossless compression schemes. The strategy behind BDM is to find small computer programs that produce the components of a larger, decomposed object. The set of short computer programs can then be artfully arranged in sequence so as to produce the original object. We show that the method provides efficient estimations of algorithmic complexity but that it performs like Shannon entropy when it loses accuracy. We estimate errors and study the behaviour of BDM for different boundary conditions, all of which are compared and assessed in detail. The measure may be adapted for use with more multi-dimensional objects than strings, objects such as arrays and tensors. To test the measure we demonstrate the power of CTM on low algorithmic-randomness objects that are assigned maximal entropy (e.g., π) but whose numerical approximations are closer to the theoretical low algorithmic-randomness expectation. We also test the measure on larger objects including dual, isomorphic and cospectral graphs for which we know that algorithmic randomness is low. We also release implementations of the methods in most major programming languages-Wolfram Language (Mathematica), Matlab, R, Perl, Python, Pascal, C++, and Haskell-and an online algorithmic complexity calculator.
dc.description.sponsorshipThis research was funded by Swedish Research Council (Vetenskapsrådet) grant number [2015-05299].
dc.eprint.versionPublisher's Version/PDF
dc.identifier.arxivid1609.00110
dc.identifier.citationZenil H, Hernández-Orozco S, Kiani N, Soler-Toscano F, Rueda-Toicen A, et al. (2018) A Decomposition Method for Global Evaluation of Shannon Entropy and Local Estimations of Algorithmic Complexity. Entropy 20: 605. Available: http://dx.doi.org/10.3390/e20080605.
dc.identifier.doi10.3390/e20080605
dc.identifier.issn1099-4300
dc.identifier.journalEntropy
dc.identifier.urihttp://hdl.handle.net/10754/628775
dc.publisherMDPI AG
dc.relation.urlhttps://www.mdpi.com/1099-4300/20/8/605
dc.rightsThis is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. (CC BY 4.0).
dc.rights.urihttps://creativecommons.org/licenses/by/4.0/
dc.subjectAlgorithmic probability
dc.subjectAlgorithmic randomness
dc.subjectInformation content
dc.subjectInformation theory
dc.subjectKolmogorov-Chaitin complexity
dc.subjectShannon entropy
dc.subjectThue-Morse sequence
dc.subjectπ
dc.titleA Decomposition Method for Global Evaluation of Shannon Entropy and Local Estimations of Algorithmic Complexity
dc.typeArticle
display.details.left<span><h5>License</h5>https://creativecommons.org/licenses/by/4.0/<br><br><h5>Type</h5>Article<br><br><h5>Authors</h5><a href="https://repository.kaust.edu.sa/search?spc.sf=dc.date.issued&spc.sd=DESC&f.author=Zenil, Hector,equals">Zenil, Hector</a><br><a href="https://repository.kaust.edu.sa/search?spc.sf=dc.date.issued&spc.sd=DESC&f.author=Hernández-Orozco, Santiago,equals">Hernández-Orozco, Santiago</a><br><a href="https://repository.kaust.edu.sa/search?spc.sf=dc.date.issued&spc.sd=DESC&f.author=Kiani, Narsis,equals">Kiani, Narsis</a><br><a href="https://repository.kaust.edu.sa/search?spc.sf=dc.date.issued&spc.sd=DESC&f.author=Soler-Toscano, Fernando,equals">Soler-Toscano, Fernando</a><br><a href="https://repository.kaust.edu.sa/search?query=orcid.id:0000-0002-6812-7337&spc.sf=dc.date.issued&spc.sd=DESC">Rueda-Toicen, Antonio</a> <a href="https://orcid.org/0000-0002-6812-7337" target="_blank"><img src="https://repository.kaust.edu.sa/server/api/core/bitstreams/82a625b4-ed4b-40c8-865a-d6a5225a26a4/content" width="16" height="16"/></a><br><a href="https://repository.kaust.edu.sa/search?query=orcid.id:0000-0002-9568-5588&spc.sf=dc.date.issued&spc.sd=DESC">Tegner, Jesper</a> <a href="https://orcid.org/0000-0002-9568-5588" target="_blank"><img src="https://repository.kaust.edu.sa/server/api/core/bitstreams/82a625b4-ed4b-40c8-865a-d6a5225a26a4/content" width="16" height="16"/></a><br><br><h5>KAUST Department</h5><a href="https://repository.kaust.edu.sa/search?spc.sf=dc.date.issued&spc.sd=DESC&f.department=Biological and Environmental Sciences and Engineering (BESE) Division,equals">Biological and Environmental Sciences and Engineering (BESE) Division</a><br><a href="https://repository.kaust.edu.sa/search?spc.sf=dc.date.issued&spc.sd=DESC&f.department=Bioscience Program,equals">Bioscience Program</a><br><a href="https://repository.kaust.edu.sa/search?spc.sf=dc.date.issued&spc.sd=DESC&f.department=Computer, Electrical and Mathematical Sciences and Engineering (CEMSE) Division,equals">Computer, Electrical and Mathematical Sciences and Engineering (CEMSE) Division</a><br><br><h5>Date</h5>2018-08-15</span>
display.details.right<span><h5>Abstract</h5>We investigate the properties of a Block Decomposition Method (BDM), which extends the power of a Coding Theorem Method (CTM) that approximates local estimations of algorithmic complexity based on Solomonoff-Levin's theory of algorithmic probability providing a closer connection to algorithmic complexity than previous attempts based on statistical regularities such as popular lossless compression schemes. The strategy behind BDM is to find small computer programs that produce the components of a larger, decomposed object. The set of short computer programs can then be artfully arranged in sequence so as to produce the original object. We show that the method provides efficient estimations of algorithmic complexity but that it performs like Shannon entropy when it loses accuracy. We estimate errors and study the behaviour of BDM for different boundary conditions, all of which are compared and assessed in detail. The measure may be adapted for use with more multi-dimensional objects than strings, objects such as arrays and tensors. To test the measure we demonstrate the power of CTM on low algorithmic-randomness objects that are assigned maximal entropy (e.g., π) but whose numerical approximations are closer to the theoretical low algorithmic-randomness expectation. We also test the measure on larger objects including dual, isomorphic and cospectral graphs for which we know that algorithmic randomness is low. We also release implementations of the methods in most major programming languages-Wolfram Language (Mathematica), Matlab, R, Perl, Python, Pascal, C++, and Haskell-and an online algorithmic complexity calculator.<br><br><h5>Citation</h5>Zenil H, Hernández-Orozco S, Kiani N, Soler-Toscano F, Rueda-Toicen A, et al. (2018) A Decomposition Method for Global Evaluation of Shannon Entropy and Local Estimations of Algorithmic Complexity. Entropy 20: 605. Available: http://dx.doi.org/10.3390/e20080605.<br><br><h5>Acknowledgements</h5>This research was funded by Swedish Research Council (Vetenskapsrådet) grant number [2015-05299].<br><br><h5>Publisher</h5><a href="https://repository.kaust.edu.sa/search?spc.sf=dc.date.issued&spc.sd=DESC&f.publisher=MDPI AG,equals">MDPI AG</a><br><br><h5>Journal</h5><a href="https://repository.kaust.edu.sa/search?spc.sf=dc.date.issued&spc.sd=DESC&f.journal=Entropy,equals">Entropy</a><br><br><h5>DOI</h5><a href="https://doi.org/10.3390/e20080605">10.3390/e20080605</a><br><br><h5>arXiv</h5><a href="https://arxiv.org/abs/1609.00110">1609.00110</a><br><br><h5>Additional Links</h5>https://www.mdpi.com/1099-4300/20/8/605</span>
kaust.personTegner, Jesper
orcid.authorZenil, Hector
orcid.authorHernández-Orozco, Santiago
orcid.authorKiani, Narsis
orcid.authorSoler-Toscano, Fernando
orcid.authorRueda-Toicen, Antonio::0000-0002-6812-7337
orcid.authorTegner, Jesper::0000-0002-9568-5588
orcid.id0000-0002-9568-5588
orcid.id0000-0002-6812-7337
refterms.dateFOA2018-09-27T07:55:38Z
Files
Original bundle
Now showing 1 - 1 of 1
Loading...
Thumbnail Image
Name:
entropy-20-00605-v2.pdf
Size:
2.01 MB
Format:
Adobe Portable Document Format
Description:
Published version