Dependency Management
Reusing dependencies across ecosystems: what stands in the way?
Every software ecosystem seems to have a package manager these days, but reusing software across these ecosystems is still a challenge. Major Linux distributions package software from a wide range of languages, but they restrict the versions you can install, and they make deep assumptions about compilers and runtime libraries to keep everything compatible. If you need a newer libc or a newer Python than the OS offers, you're often on your own. Python packaging supports native libraries, but it imposes strict rules on package builders to ensure that the binaries work in many places. Some packagers break the rules, e.g., TensorFlow binary packages (wheels) will only work on Ubuntu, frustrating users of other distributions. Containers provide some hope, but only if you stay inside their sandbox. If you your container to talk to a fast network or a GPU, you'll need to ensure binary compatibility between the container and the host OS. Is every ecosystem a walled garden or can we make it easier to bridge the gap?
This talk takes a deeper look at the Application Binary Interface (ABI) and its place at the heart of nearly every software ecosystem. We'll look at the assumptions made by modern dependency management systems about toolchains and ABI. Most tools assume that toolchains and ABI are fixed, making it hard to take a package from one ecosystem and transplant it in another. We'll talk about how we can build dependency managers that can reason better about ABI compatibility, making decisions not just about which software versions to install but also about how those software versions should be built. We'll look at the types of metadata that need to be managed to enable a more flexible software environment, and how it affects the complexity of dependency resolution. Finally, we'll look at how some of these issues are being addressed in Spack, an open source package manager for high performance computing applications, which can span multiple languages, compilers, platforms, and software stacks.
Additional information
| Type | devroom |
|---|
More sessions
| 2/7/21 |
The goal of the EU project FASTEN is being able to perform a more sophisticated analysis of security-vulnerability propagation, licensing compliance, and dependency risk profiles (among others) by relying on the call-level dependency network of the whole software ecosystem. We outline the purpose and structure of the project, and present some preliminary results.
|
| 2/7/21 |
The talk introduces DepClean, an open-source tool that we developed to automatically determine the presence of bloated dependencies in Maven artifacts. DepClean performs a deep static analysis of the dependency network and suggests direct and transitive dependencies to be removed or excluded. Given an application and its build file, DepClean collects the complete dependency tree (the list of dependencies declared in the pom.xml, as well as the transitive dependencies) and analyzes the bytecode ...
|
| 2/7/21 |
When developing open source software end-user applications or reusable software packages, developers depend on software packages distributed through package managers such as npm, Packagist, Cargo, RubyGems. In addition to this, empirical evidence has shown that these package managers adhere to a large extent to semantic versioning principles. Packages that are still in major version zero are considered unstable according to semantic versioning, as some developers consider such packages as ...
|
| 2/7/21 |
Despite best intentions, Open Source releases with regression errors are published every day. In the best case scenario, a downstream user detects it early thanks to good tests, files an issue, and the maintainer can fix it before too many people have upgraded. Other scenarios involve various degrees of brokenness and games of "is it broken for everyone or just me?". Renovate Bot is an open source dependency automation tool but which also is run as a free app on github.com, where it is installed ...
|
| 2/7/21 |
The Solarwinds breach at the end of 2020 is an event that we won't truly understand the breadth and depth of for some time - if ever. But already, several discussions we've been having in the abstract for years have become very concrete. Firstly, the systems we use to develop, code, build and deploy our code are all essential production systems - and should be treated as such. And second, securing the software supply chain is one of the most underrated aspects of security and is often ...
|
