Robust and Scalable Sampling Algorithms for Network Measurement PDF Download

Author: Xiaoming Wang
Publisher:
ISBN:
Category :
Languages : en
Pages :

View

Book Description
Recent growth of the Internet in both scale and complexity has imposed a number of difficult challenges on existing measurement techniques and approaches, which are essential for both network management and many ongoing research projects. For any measurement algorithm, achieving both accuracy and scalability is very challenging given hard resource constraints (e.g., bandwidth, delay, physical memory, and CPU speed). My dissertation research tackles this problem by first proposing a novel mechanism called residual sampling, which intentionally introduces a predetermined amount of bias into the measurement process. We show that such biased sampling can be extremely scalable; moreover, we develop residual estimation algorithms that can unbiasedly recover the original information from the sampled data. Utilizing these results, we further develop two versions of the residual sampling mechanism: a continuous version for characterizing the user lifetime distribution in large-scale peer-to-peer networks and a discrete version for monitoring flow statistics (including per-flow counts and the flow size distribution) in high-speed Internet routers. For the former application in P2P networks, this work presents two methods: ResIDual-based Estimator (RIDE), which takes single-point snapshots of the system and assumes systems with stationary arrivals, and Uniform RIDE (U-RIDE), which takes multiple snapshots and adapts to systems with arbitrary (including non-stationary) arrival processes. For the latter application in traffic monitoring, we introduce Discrete RIDE (D-RIDE), which allows one to sample each flow with a geometric random variable. Our numerous simulations and experiments with P2P networks and real Internet traces confirm that these algorithms are able to make accurate estimation about the monitored metrics and simultaneously meet the requirements of hard resource constraints. These results show that residual sampling indeed provides an ideal solution to balancing between accuracy and scalability.

Author: Shang-Hua Teng
Publisher:
ISBN: 9781680831306
Category : Computers
Languages : en
Pages : 292

View

Book Description
In the age of Big Data, efficient algorithms are in high demand. It is also essential that efficient algorithms should be scalable. This book surveys a family of algorithmic techniques for the design of scalable algorithms. These techniques include local network exploration, advanced sampling, sparsification, and geometric partitioning.

Author: Feizi-Khankandi Feizi
Publisher:
ISBN:
Category :
Languages : en
Pages : 496

View

Book Description
Network models provide a unifying framework for understanding dependencies among variables in data-driven and engineering sciences. Networks can be used to reveal underlying data structures, infer functional modules, and facilitate experiment design. In practice, however, size, uncertainty and complexity of the underlying associations render these applications challenging. In this thesis, we illustrate the use of spectral, combinatorial, and statistical inference techniques in several network science problems. In Chapters 2-4, we consider network inference challenges. In Chapter 2, we introduce Network Maximal Correlation (NMC) as a multivariate measure of nonlinear association suitable for evaluation on large datasets. We characterize a solution of the NMC optimization using geometric properties of Hilbert spaces for finite discrete and jointly Gaussian random variables. We illustrate an application of NMC and multiple MC in inference of graphical models for bijective, possibly non-monotone, functions of jointly Gaussian variables. As a demonstration of NMC's utility, we infer nonlinear gene association networks and modules in cancer datasets and validate them using survival times of patients. In Chapter 3, we develop a network integration framework to infer gene regulatory networks in human and model organisms fly and worm using diverse and high-throughput datasets. Inferred regulatory interactions have significant overlap with known edges, indicating the robustness and accuracy of the proposed network inference framework. In Chapter 4, we formulate the transitive noise problem in networks as the inverse of matrix transitive closure and introduce an algorithm to solve it efficiently. We demonstrate the effectiveness of our approach in several applications such as regulatory network inference, protein contact map inference and strong collaboration tie inference. In Chapters 5-8, we consider network analysis challenges. In Chapter 5, we consider the problem of network alignment where the goal is to find a bijective mapping between nodes of two networks to maximize their overlapping edges while minimizing mismatches. This problem is essential in comparative analysis across large datasets and networks. To solve this combinatorial problem, we present a new scalable spectral algorithm which creates an eigenvector relaxation for the underlying optimization. We prove the optimality of the method under certain technical conditions, and show its effectiveness over various synthetic networks as well as in comparative analysis of gene regulatory networks across human, fly and worm species. In Chapter 6, we consider the source inference problem where the goal is to identify the source(s) of propagated signals across biological, social and engineered networks. To solve this problem, we propose a computationally tractable general method based on a path-based network diffusion kernel. We prove mean-field optimality of this method for different scenarios and show its effectiveness over several synthetic networks as well as in identifying sources in a Digg social news network. In Chapter 7, we consider the problem of learning low dimensional structures (such as clusters) in large networks. Here we introduce logistic Random Dot Product Graphs (RDPGs) as a new class of networks which includes most stochastic block models as well as other low dimensional structures. Using this model, we propose a scalable spectral method that solves the maximum likelihood inference problem asymptotically exactly. This leads to a new scalable spectral network clustering algorithm that is robust under different clustering setups. In Chapter 8, we consider the biclustering problem, the analog of clustering on bipartite graphs. This problem has several applications such as inference of co-regulated genes, document classification, and so on. Here we propose an algorithm based on message-passing that closely approximates a general likelihood function and excels at resolving the overlaps between biclusters. In Chapters 9-12, we consider design challenges of systems and algorithms for engineering networks such as communication networks. In Chapters 9-10, we create a connection between compressive sensing and traditional information theoretic techniques in source, channel and network coding and propose a joint coding scheme over wireless networks based on random projection and restricted eigenvalue principles. Moreover, we characterize fundamental results on the trade-off between the communication rate and the decoding complexity. In Chapters 11-12, we propose an adaptive nonuniform sampling framework, in which time increments between samples are determined as a function of the most recent increments and sample values, obviating the need to track time stamps. We analyze the performance of the proposed method for different stochastic and deterministic signal models and show its effectiveness to enhance measurements of heart ECG signals.

Author: Armin Biere
Publisher: Springer Nature
ISBN: 3030452379
Category : Computers
Languages : en
Pages : 425

View

Book Description
This open access two-volume set constitutes the proceedings of the 26th International Conference on Tools and Algorithms for the Construction and Analysis of Systems, TACAS 2020, which took place in Dublin, Ireland, in April 2020, and was held as Part of the European Joint Conferences on Theory and Practice of Software, ETAPS 2020. The total of 60 regular papers presented in these volumes was carefully reviewed and selected from 155 submissions. The papers are organized in topical sections as follows: Part I: Program verification; SAT and SMT; Timed and Dynamical Systems; Verifying Concurrent Systems; Probabilistic Systems; Model Checking and Reachability; and Timed and Probabilistic Systems. Part II: Bisimulation; Verification and Efficiency; Logic and Proof; Tools and Case Studies; Games and Automata; and SV-COMP 2020.

Author: Baek-Young Choi
Publisher: Springer Science & Business Media
ISBN: 1461401194
Category : Computers
Languages : en
Pages : 161

View

Book Description
Network monitoring serves as the basis for a wide scope of network, engineering and management operations. Precise network monitoring involves inspecting every packet traversing in a network. However, this is not feasible with future high-speed networks, due to significant overheads of processing, storing, and transferring measured data. Network Monitoring in High Speed Networks presents accurate measurement schemes from both traffic and performance perspectives, and introduces adaptive sampling techniques for various granularities of traffic measurement. The techniques allow monitoring systems to control the accuracy of estimations, and adapt sampling probability dynamically according to traffic conditions. The issues surrounding network delays for practical performance monitoring are discussed in the second part of this book. Case studies based on real operational network traces are provided throughout this book. Network Monitoring in High Speed Networks is designed as a secondary text or reference book for advanced-level students and researchers concentrating on computer science and electrical engineering. Professionals working within the networking industry will also find this book useful.

Author: Bernhard Steffen
Publisher: Springer Nature
ISBN: 3031460022
Category : Computers
Languages : en
Pages : 454

View

Book Description
This book constitutes the proceedings of the First International Conference on Bridging the Gap between AI and Reality, AISoLA 2023, which took place in Crete, Greece, in October 2023. The papers included in this book focus on the following topics: The nature of AI-based systems; ethical, economic and legal implications of AI-systems in practice; ways to make controlled use of AI via the various kinds of formal methods-based validation techniques; dedicated applications scenarios which may allow certain levels of assistance; and education in times of deep learning.

Author: Michael Simpson
Publisher:
ISBN:
Category :
Languages : en
Pages :

View

Book Description
This thesis investigates several problems in social network analysis on misinformation prevention with an emphasis on finding solutions that can scale to massive online networks. In particular, it considers two problem formulations related to the spread of misinformation in a network that cover the elimination of existing misinformation and the prevention of future dissemination of misinformation. Additionally, a comprehensive comparison of several algorithms for the feedback arc set (FAS) problem is presented in order to identify an approach that is both scalable and computes a lightweight solution. The feedback arc set problem is of particular interest since several notable problems in social network analysis, including the elimination of existing misinformation, crucially rely on computing a small FAS as a preliminary. The elimination of existing misinformation is modelled as a graph searching game. The problem can be summarized as constructing a search strategy that will leave the graph clear of any misinformation at the end of the searching process in as few steps as possible. Despite the problem being NP-hard, even on directed acyclic graphs, this thesis presents an efficient approximation algorithm and provides new experimental results that compares the performance of the approximation algorithm to the lower bound on several large online networks. In particular, new scalability goals are achieved through careful algorithmic engineering and a highly optimized pre-processing step. The minimum feedback arc set problem is an NP-hard problem on graphs that seeks a minimum set of arcs which, when removed from the graph, leave it acyclic. A comprehensive comparison of several approximation algorithms for computing a minimum feedback arc set is presented with the goal of comparing the quality of the solutions and the running times. Additionally, careful algorithmic engineering is applied for multiple algorithms in order to improve their scalability. In particular, two approaches that are optimized (one greedy and one randomized) result in simultaneously strong performance for both feedback arc set size and running time. The experiments compare the performance of a wide range of algorithms on a broad selection of large online networks and reveal that the optimized greedy and randomized implementations outperform the other approaches by simultaneously computing a feedback arc set of competitive size and scaling to web-scale graphs with billions of vertices and tens of billions of arcs. Finally, the algorithms considered are extended to the probabilistic case in which arcs are realized with some fixed probability and a detailed experimental comparison is provided. \sloppy Finally, the problem of preventing the spread of misinformation propagating through a social network is considered. In this problem, a ``bad'' campaign starts propagating from a set of seed nodes in the network and the notion of a limiting (or ``good'') campaign is used to counteract the effect of misinformation. The goal is to identify a set of $k$ users that need to be convinced to adopt the limiting campaign so as to minimize the number of people that adopt the ``bad'' campaign at the end of both propagation processes. \emph{RPS} (Reverse Prevention Sampling), an algorithm that provides a scalable solution to the misinformation prevention problem, is presented. The theoretical analysis shows that \emph{RPS} runs in $O((k + l)(n + m)(\frac{1}{1 - \gamma}) \log n / \epsilon^2 )$ expected time and returns a $(1 - 1/e - \epsilon)$-approximate solution with at least $1 - n^{-l}$ probability (where $\gamma$ is a typically small network parameter). The time complexity of \emph{RPS} substantially improves upon the previously best-known algorithms that run in time $\Omega(m n k \cdot POLY(\epsilon^{-1}))$. Additionally, an experimental evaluation of \emph{RPS} on large datasets is presented where it is shown that \emph{RPS} outperforms the state-of-the-art solution by several orders of magnitude in terms of running time. This demonstrates that misinformation prevention can be made practical while still offering strong theoretical guarantees.

Author: Mohamad Bdiwi
Publisher: Frontiers Media SA
ISBN: 2832523927
Category : Technology & Engineering
Languages : en
Pages : 127

View

Book Description

Author: International Joint Conferences on Artificial Intelligence
Publisher: Morgan Kaufmann
ISBN: 9781558604803
Category : Artificial intelligence
Languages : en
Pages : 1720

View

Book Description

Author: Xingming Sun
Publisher: Springer
ISBN: 3030242684
Category : Computers
Languages : en
Pages : 651

View

Book Description
The 4-volume set LNCS 11632 until LNCS 11635 constitutes the refereed proceedings of the 5th International Conference on Artificial Intelligence and Security, ICAIS 2019, which was held in New York, USA, in July 2019. The conference was formerly called “International Conference on Cloud Computing and Security” with the acronym ICCCS. The total of 230 full papers presented in this 4-volume proceedings was carefully reviewed and selected from 1529 submissions. The papers were organized in topical sections as follows: Part I: cloud computing; Part II: artificial intelligence; big data; and cloud computing and security; Part III: cloud computing and security; information hiding; IoT security; multimedia forensics; and encryption and cybersecurity; Part IV: encryption and cybersecurity.