Large-scale Statistical Inference for Graph-associated Data PDF Download

Author: Tien Vo
Publisher:
ISBN:
Category :
Languages : en
Pages : 0

View

Book Description
Large-scale hypothesis testing is very important for assessing population differences from sampled data in various application domains. In many cases, high-dimensional data are naturally associated with a graphical architecture, in which measured variables reside on graph vertices and the connectivity of the graph conveys information about the underlying relational structure among the data. Essentially, each edge in the graph represents the relationship between values at its endpoints due to some conceptual dependency, e.g temporal, spatial, functional, anatomical, etc. Available large-scale testing methods often consider dependencies a nuisance, and, by using sufficiently simple, unit-level test statistics, aim to control false discovery rate in a way that is robust to details of such dependence. Where some available methods do incorporate models of dependence, they are limited in scope and they do not take advantage of the graphical nature of the data structure. Given shortcomings of available methods and the importance of the largescale testing problem, we propose a new methodology to incorporate graphical information for hypothesis testing. Our proposed method, graph-based mixture model (GraphMM) is a semiparametric empirical Bayesian approach, motivated from a hybrid procedure that exploits grouping information of model parameters to increase testing sensitivity. We conduct experiments on a parallel computing platform and apply model in the context of a neuroimaging task to detect subtle changes from magnetic resonance imagery.

Author: Bradley Efron
Publisher: Cambridge University Press
ISBN: 1139492136
Category : Mathematics
Languages : en
Pages :

View

Book Description
We live in a new age for statistical inference, where modern scientific technology such as microarrays and fMRI machines routinely produce thousands and sometimes millions of parallel data sets, each with its own estimation or testing problem. Doing thousands of problems at once is more than repeated application of classical methods. Taking an empirical Bayes approach, Bradley Efron, inventor of the bootstrap, shows how information accrues across problems in a way that combines Bayesian and frequentist ideas. Estimation, testing and prediction blend in this framework, producing opportunities for new methodologies of increased power. New difficulties also arise, easily leading to flawed inferences. This book takes a careful look at both the promise and pitfalls of large-scale statistical inference, with particular attention to false discovery rates, the most successful of the new statistical techniques. Emphasis is on the inferential ideas underlying technical developments, illustrated using a large number of real examples.

Author: National Research Council
Publisher: National Academies Press
ISBN: 0309287812
Category : Mathematics
Languages : en
Pages : 191

View

Book Description
Data mining of massive data sets is transforming the way we think about crisis response, marketing, entertainment, cybersecurity and national intelligence. Collections of documents, images, videos, and networks are being thought of not merely as bit strings to be stored, indexed, and retrieved, but as potential sources of discovery and knowledge, requiring sophisticated analysis techniques that go far beyond classical indexing and keyword counting, aiming to find relational and semantic interpretations of the phenomena underlying the data. Frontiers in Massive Data Analysis examines the frontier of analyzing massive amounts of data, whether in a static database or streaming through a system. Data at that scale-terabytes and petabytes-is increasingly common in science (e.g., particle physics, remote sensing, genomics), Internet commerce, business analytics, national security, communications, and elsewhere. The tools that work to infer knowledge from data at smaller scales do not necessarily work, or work well, at such massive scale. New tools, skills, and approaches are necessary, and this report identifies many of them, plus promising research directions to explore. Frontiers in Massive Data Analysis discusses pitfalls in trying to infer knowledge from massive data, and it characterizes seven major classes of computation that are common in the analysis of massive data. Overall, this report illustrates the cross-disciplinary knowledge-from computer science, statistics, machine learning, and application disciplines-that must be brought to bear to make useful inferences from massive data.

Author: National Research Council
Publisher: National Academies Press
ISBN: 0309287782
Category : Mathematics
Languages : en
Pages : 191

View

Book Description
Data mining of massive data sets is transforming the way we think about crisis response, marketing, entertainment, cybersecurity and national intelligence. Collections of documents, images, videos, and networks are being thought of not merely as bit strings to be stored, indexed, and retrieved, but as potential sources of discovery and knowledge, requiring sophisticated analysis techniques that go far beyond classical indexing and keyword counting, aiming to find relational and semantic interpretations of the phenomena underlying the data. Frontiers in Massive Data Analysis examines the frontier of analyzing massive amounts of data, whether in a static database or streaming through a system. Data at that scale-terabytes and petabytes-is increasingly common in science (e.g., particle physics, remote sensing, genomics), Internet commerce, business analytics, national security, communications, and elsewhere. The tools that work to infer knowledge from data at smaller scales do not necessarily work, or work well, at such massive scale. New tools, skills, and approaches are necessary, and this report identifies many of them, plus promising research directions to explore. Frontiers in Massive Data Analysis discusses pitfalls in trying to infer knowledge from massive data, and it characterizes seven major classes of computation that are common in the analysis of massive data. Overall, this report illustrates the cross-disciplinary knowledge-from computer science, statistics, machine learning, and application disciplines-that must be brought to bear to make useful inferences from massive data.

Author: S. Ejaz Ahmed
Publisher: Springer
ISBN: 3319415735
Category : Mathematics
Languages : en
Pages : 390

View

Book Description
This volume conveys some of the surprises, puzzles and success stories in high-dimensional and complex data analysis and related fields. Its peer-reviewed contributions showcase recent advances in variable selection, estimation and prediction strategies for a host of useful models, as well as essential new developments in the field. The continued and rapid advancement of modern technology now allows scientists to collect data of increasingly unprecedented size and complexity. Examples include epigenomic data, genomic data, proteomic data, high-resolution image data, high-frequency financial data, functional and longitudinal data, and network data. Simultaneous variable selection and estimation is one of the key statistical problems involved in analyzing such big and complex data. The purpose of this book is to stimulate research and foster interaction between researchers in the area of high-dimensional data analysis. More concretely, its goals are to: 1) highlight and expand the breadth of existing methods in big data and high-dimensional data analysis and their potential for the advancement of both the mathematical and statistical sciences; 2) identify important directions for future research in the theory of regularization methods, in algorithmic development, and in methodologies for different application areas; and 3) facilitate collaboration between theoretical and subject-specific researchers.

Author: Murray Aitkin
Publisher: CRC Press
ISBN: 100064457X
Category : Mathematics
Languages : en
Pages : 391

View

Book Description
The complexity of large-scale data sets (“Big Data”) has stimulated the development of advanced computational methods for analysing them. There are two different kinds of methods to aid this. The model-based method uses probability models and likelihood and Bayesian theory, while the model-free method does not require a probability model, likelihood or Bayesian theory. These two approaches are based on different philosophical principles of probability theory, espoused by the famous statisticians Ronald Fisher and Jerzy Neyman. Introduction to Statistical Modelling and Inference covers simple experimental and survey designs, and probability models up to and including generalised linear (regression) models and some extensions of these, including finite mixtures. A wide range of examples from different application fields are also discussed and analysed. No special software is used, beyond that needed for maximum likelihood analysis of generalised linear models. Students are expected to have a basic mathematical background in algebra, coordinate geometry and calculus. Features • Probability models are developed from the shape of the sample empirical cumulative distribution function (cdf) or a transformation of it. • Bounds for the value of the population cumulative distribution function are obtained from the Beta distribution at each point of the empirical cdf. • Bayes’s theorem is developed from the properties of the screening test for a rare condition. • The multinomial distribution provides an always-true model for any randomly sampled data. • The model-free bootstrap method for finding the precision of a sample estimate has a model-based parallel – the Bayesian bootstrap – based on the always-true multinomial distribution. • The Bayesian posterior distributions of model parameters can be obtained from the maximum likelihood analysis of the model. This book is aimed at students in a wide range of disciplines including Data Science. The book is based on the model-based theory, used widely by scientists in many fields, and compares it, in less detail, with the model-free theory, popular in computer science, machine learning and official survey analysis. The development of the model-based theory is accelerated by recent developments in Bayesian analysis.

Author: Bradley Efron
Publisher: Cambridge University Press
ISBN: 1108915876
Category : Mathematics
Languages : en
Pages : 514

View

Book Description
The twenty-first century has seen a breathtaking expansion of statistical methodology, both in scope and influence. 'Data science' and 'machine learning' have become familiar terms in the news, as statistical methods are brought to bear upon the enormous data sets of modern science and commerce. How did we get here? And where are we going? How does it all fit together? Now in paperback and fortified with exercises, this book delivers a concentrated course in modern statistical thinking. Beginning with classical inferential theories - Bayesian, frequentist, Fisherian - individual chapters take up a series of influential topics: survival analysis, logistic regression, empirical Bayes, the jackknife and bootstrap, random forests, neural networks, Markov Chain Monte Carlo, inference after model selection, and dozens more. The distinctly modern approach integrates methodology and algorithms with statistical inference. Each chapter ends with class-tested exercises, and the book concludes with speculation on the future direction of statistics and data science.

Author: Peter Hussami
Publisher: LAP Lambert Academic Publishing
ISBN: 9783848426416
Category :
Languages : en
Pages : 100

View

Book Description
The study of random graphs is a field that emerged in the second half of the 20th century. Most of the work in this area is combinatorial in nature: a random model is assumed and used for computing various asymptotic properties on the graph. The work contained in this book takes a reverse approach. Our questions are: given a large graph realization, what can we learn about it? How can we hypothesize an underlying model? How can we test the graph for the hypothesis? The first half of the book is a survey of some well-known and less well-known methods, and apply some of them on various "scale-free" graphs, including the famous Albert-Barabási graph model. Then we proceed beyond the scale-free realm to examine the problem of generation uniformly distributed graphs with a given expected degree sequence. We hope every reader will find something to enjoy in and/or learn from the book.

Author: Chester Ismay
Publisher: CRC Press
ISBN: 1000763463
Category : Mathematics
Languages : en
Pages : 461

View

Book Description
Statistical Inference via Data Science: A ModernDive into R and the Tidyverse provides a pathway for learning about statistical inference using data science tools widely used in industry, academia, and government. It introduces the tidyverse suite of R packages, including the ggplot2 package for data visualization, and the dplyr package for data wrangling. After equipping readers with just enough of these data science tools to perform effective exploratory data analyses, the book covers traditional introductory statistics topics like confidence intervals, hypothesis testing, and multiple regression modeling, while focusing on visualization throughout. Features: ● Assumes minimal prerequisites, notably, no prior calculus nor coding experience ● Motivates theory using real-world data, including all domestic flights leaving New York City in 2013, the Gapminder project, and the data journalism website, FiveThirtyEight.com ● Centers on simulation-based approaches to statistical inference rather than mathematical formulas ● Uses the infer package for "tidy" and transparent statistical inference to construct confidence intervals and conduct hypothesis tests via the bootstrap and permutation methods ● Provides all code and output embedded directly in the text; also available in the online version at moderndive.com This book is intended for individuals who would like to simultaneously start developing their data science toolbox and start learning about the inferential and modeling tools used in much of modern-day research. The book can be used in methods and data science courses and first courses in statistics, at both the undergraduate and graduate levels.

Author: Sherif Sakr
Publisher: Springer
ISBN: 3319735152
Category : Computers
Languages : en
Pages : 236

View

Book Description
This book describes efficient and effective techniques for harnessing the power of Linked Data by tackling the various aspects of managing its growing volume: storing, querying, reasoning, provenance management and benchmarking. To this end, Chapter 1 introduces the main concepts of the Semantic Web and Linked Data and provides a roadmap for the book. Next, Chapter 2 briefly presents the basic concepts underpinning Linked Data technologies that are discussed in the book. Chapter 3 then offers an overview of various techniques and systems for centrally querying RDF datasets, and Chapter 4 outlines various techniques and systems for efficiently querying large RDF datasets in distributed environments. Subsequently, Chapter 5 explores how streaming requirements are addressed in current, state-of-the-art RDF stream data processing. Chapter 6 covers performance and scaling issues of distributed RDF reasoning systems, while Chapter 7 details benchmarks for RDF query engines and instance matching systems. Chapter 8 addresses the provenance management for Linked Data and presents the different provenance models developed. Lastly, Chapter 9 offers a brief summary, highlighting and providing insights into some of the open challenges and research directions. Providing an updated overview of methods, technologies and systems related to Linked Data this book is mainly intended for students and researchers who are interested in the Linked Data domain. It enables students to gain an understanding of the foundations and underpinning technologies and standards for Linked Data, while researchers benefit from the in-depth coverage of the emerging and ongoing advances in Linked Data storing, querying, reasoning, and provenance management systems. Further, it serves as a starting point to tackle the next research challenges in the domain of Linked Data management.