Friday 27 April 2018

Package Management Basics: apt, yum, dnf, pkg


Package Management Basics: apt, yum, dnf, pkg


Introduction

Most modern Unix-like operating systems offer a centralized mechanism for finding and installing software. Software is usually distributed in the form of packages, kept in repositories. Working with packages is known as package management. Packages provide the basic components of an operating system, along with shared libraries, applications, services, and documentation.
A package management system does much more than one-time installation of software. It also provides tools for upgrading already-installed packages. Package repositories help to ensure that code has been vetted for use on your system, and that the installed versions of software have been approved by developers and package maintainers.
When configuring servers or development environments, it's often necessary look beyond official repositories. Packages in the stable release of a distribution may be out of date, especially where new or rapidly-changing software is concerned. Nevertheless, package management is a vital skill for system administrators and developers, and the wealth of packaged software for major distributions is a tremendous resource.
This guide is intended as a quick reference for the fundamentals of finding, installing, and upgrading packages on a variety of distributions, and should help you translate that knowledge between systems.

Package Management Systems: A Brief Overview

Most package systems are built around collections of package files. A package file is usually an archive which contains compiled binaries and other resources making up the software, along with installation scripts. Packages also contain valuable metadata, including their dependencies, a list of other packages required to install and run them.
While their functionality and benefits are broadly similar, packaging formats and tools vary by platform:
Operating SystemFormatTool(s)
Debian.debaptapt-cacheapt-getdpkg
Ubuntu.debaptapt-cacheapt-getdpkg
CentOS.rpmyum
Fedora.rpmdnf
FreeBSDPorts, .txzmakepkg
In Debian and systems based on it, like Ubuntu, Linux Mint, and Raspbian, the package format is the .debfile. APT, the Advanced Packaging Tool, provides commands used for most common operations: Searching repositories, installing collections of packages and their dependencies, and managing upgrades. APT commands operate as a front-end to the lower-level dpkg utility, which handles the installation of individual .deb files on the local system, and is sometimes invoked directly.
Recent releases of most Debian-derived distributions include the apt command, which offers a concise and unified interface to common operations that have traditionally been handled by the more-specific apt-get and apt-cache. Its use is optional, but may simplify some tasks.
CentOS, Fedora, and other members of the Red Hat family use RPM files. In CentOS, yum is used to interact with both individual package files and repositories.
In recent versions of Fedora, yum has been supplanted by dnf, a modernized fork which retains most of yum's interface.
FreeBSD's binary package system is administered with the pkg command. FreeBSD also offers the Ports Collection, a local directory structure and tools which allow the user to fetch, compile, and install packages directly from source using Makefiles. It's usually much more convenient to use pkg, but occasionally a pre-compiled package is unavailable, or you may need to change compile-time options.

Update Package Lists

Most systems keep a local database of the packages available from remote repositories. It's best to update this database before installing or upgrading packages. As a partial exception to this pattern, yumand dnf will check for updates before performing some operations, but you can ask them at any time whether updates are available.
SystemCommand
Debian / Ubuntusudo apt-get update
sudo apt update
CentOSyum check-update
Fedoradnf check-update
FreeBSD Packagessudo pkg update
FreeBSD Portssudo portsnap fetch update

Upgrade Installed Packages

Making sure that all of the installed software on a machine stays up to date would be an enormous undertaking without a package system. You would have to track upstream changes and security alerts for hundreds of different packages. While a package manager doesn't solve every problem you'll encounter when upgrading software, it does enable you to maintain most system components with a few commands.
On FreeBSD, upgrading installed ports can introduce breaking changes or require manual configuration steps. It's best to read /usr/ports/UPDATING before upgrading with portmaster.
SystemCommandNotes
Debian / Ubuntusudo apt-get upgradeOnly upgrades installed packages, where possible.
sudo apt-get dist-upgradeMay add or remove packages to satisfy new dependencies.
sudo apt upgradeLike apt-get upgrade.
sudo apt full-upgradeLike apt-get dist-upgrade.
CentOSsudo yum update
Fedorasudo dnf upgrade
FreeBSD Packagessudo pkg upgrade
FreeBSD Portsless /usr/ports/UPDATINGUses less to view update notes for ports (use arrow keys to scroll, press qto quit).
cd /usr/ports/ports-mgmt/portmaster && sudo make install && sudo portmaster -aInstalls portmaster and uses it to update installed ports.

Find a Package

Most distributions offer a graphical or menu-driven front end to package collections. These can be a good way to browse by category and discover new software. Often, however, the quickest and most effective way to locate a package is to search with command-line tools.
SystemCommandNotes
Debian / Ubuntuapt-cache search search_string
apt search search_string
CentOSyum search search_string
yum search all search_stringSearches all fields, including description.
Fedoradnf search search_string
dnf search all search_stringSearches all fields, including description.
FreeBSD Packagespkg search search_stringSearches by name.
pkg search -f search_stringSearches by name, returning full descriptions.
pkg search -D search_stringSearches description.
FreeBSD Portscd /usr/ports && make search name=packageSearches by name.
cd /usr/ports && make search key=search_stringSearches comments, descriptions, and dependencies.

View Info About a Specific Package

When deciding what to install, it's often helpful to read detailed descriptions of packages. Along with human-readable text, these often include metadata like version numbers and a list of the package's dependencies.
SystemCommandNotes
Debian / Ubuntuapt-cache show packageShows locally-cached info about a package.
apt show package
dpkg -s packageShows the current installed status of a package.
CentOSyum info package
yum deplist packageLists dependencies for a package.
Fedoradnf info package
dnf repoquery --requires packageLists dependencies for a package.
FreeBSD Packagespkg info packageShows info for an installed package.
FreeBSD Portscd /usr/ports/category/port && cat pkg-descr

Install a Package from Repositories

Once you know the name of a package, you can usually install it and its dependencies with a single command. In general, you can supply multiple packages to install simply by listing them all.
SystemCommandNotes
Debian / Ubuntusudo apt-get install package
sudo apt-get install package1 package2 ...Installs all listed packages.
sudo apt-get install -y packageAssumes "yes" where apt would usually prompt to continue.
sudo apt install packageDisplays a colored progress bar.
CentOSsudo yum install package
sudo yum install package1 package2 ...Installs all listed packages.
sudo yum install -y packageAssumes "yes" where yum would usually prompt to continue.
Fedorasudo dnf install package
sudo dnf install package1 package2 ...Installs all listed packages.
sudo dnf install -y packageAssumes "yes" where dnf would usually prompt to continue.
FreeBSD Packagessudo pkg install package
sudo pkg install package1 package2 ...Installs all listed packages.
FreeBSD Portscd /usr/ports/category/port && sudo make installBuilds and installs a port from source.

Install a Package from the Local Filesystem

Sometimes, even though software isn't officially packaged for a given operating system, a developer or vendor will offer package files for download. You can usually retrieve these with your web browser, or via curl on the command line. Once a package is on the target system, it can often be installed with a single command.
On Debian-derived systems, dpkg handles individual package files. If a package has unmet dependencies, gdebi can often be used to retrieve them from official repositories.
On CentOS and Fedora systems, yum and dnf are used to install individual files, and will also handle needed dependencies.
SystemCommandNotes
Debian / Ubuntusudo dpkg -i package.deb
sudo apt-get install -y gdebi&& sudo gdebi package.debInstalls and uses gdebi to install package.deband retrieve any missing dependencies.
CentOSsudo yum install package.rpm
Fedorasudo dnf install package.rpm
FreeBSD Packagessudo pkg add package.txz
sudo pkg add -f package.txzInstalls package even if already installed.

Remove One or More Installed Packages

Since a package manager knows what files are provided by a given package, it can usually remove them cleanly from a system if the software is no longer needed.
SystemCommandNotes
Debian / Ubuntusudo apt-get remove package
sudo apt remove package
sudo apt-get autoremoveRemoves unneeded packages.
CentOSsudo yum remove package
Fedorasudo dnf erase package
FreeBSD Packagessudo pkg delete package
sudo pkg autoremoveRemoves unneeded packages.
FreeBSD Portssudo pkg delete package
cd /usr/ports/path_to_port && make deinstallDe-installs an installed port.

The apt Command

Administrators of Debian-family distributions are generally familiar with apt-get and apt-cache. Less widely known is the simplified apt interface, designed specifically for interactive use.
Traditional Commandapt Equivalent
apt-get updateapt update
apt-get dist-upgradeapt full-upgrade
apt-cache search stringapt search string
apt-get install packageapt install package
apt-get remove packageapt remove package
apt-get purge packageapt purge package
While apt is often a quicker shorthand for a given operation, it's not intended as a complete replacement for the traditional tools, and its interface may change between versions to improve usability. If you are using package management commands inside a script or a shell pipeline, it's a good idea to stick with apt-get and apt-cache.

Get Help

In addition to web-based documentation, keep in mind that Unix manual pages (usually referred to as man pages) are available for most commands from the shell. To read a page, use man:
  • man page
In man, you can navigate with the arrow keys. Press / to search for text within the page, and q to quit.
SystemCommandNotes
Debian / Ubuntuman apt-getUpdating the local package database and working with packages.
man apt-cacheQuerying the local package database.
man dpkgWorking with individual package files and querying installed packages.
man aptWorking with a more concise, user-friendly interface to most basic operations.
CentOSman yum
Fedoraman dnf
FreeBSD Packagesman pkgWorking with pre-compiled binary packages.
FreeBSD Portsman portsWorking with the Ports Collection.

Conclusion and Further Reading

Thursday 5 April 2018

AWS Fargate

Introducing AWS Fargate – Run Containers without Managing Infrastructure



Containers are a powerful way for developers to develop, package, and deploy their applications. At AWS we have over a hundred thousand active ECS clusters and hundreds of millions of new containers started each week. That’s 400+% customer growth since 2016. Container orchestration solutions, like Amazon ECS and Kubernetes make it easier to deploy, manage, and scale these container workloads increasing your agility. However, with each of these container management solutions you’re still responsible for the availability, capacity, and maintenance of the underlying infrastructure. At AWS we saw this as an opportunity to remove some undifferentiated heavy lifting. We want to let you take full advantage of the speed, agility, and immutability that containers offer so you can focus on building your applications rather than managing your infrastructure.

AWS Fargate

AWS Fargate is an easy way to deploy your containers on AWS. To put it simply, Fargate is like EC2 but instead of giving you a virtual machine you get a container. It’s a technology that allows you to use containers as a fundamental compute primitive without having to manage the underlying instances. All you need to do is build your container image, specify the CPU and memory requirements, define your networking and IAM policies, and launch. With Fargate, you have flexible configuration options to closely match your application needs and you’re billed with per-second granularity.
The best part? You can still use all of the same ECS primitives, APIs, and AWS integrations. Fargate provides native integrations with Amazon Virtual Private CloudAWS Identity and Access Management (IAM)Amazon CloudWatch and load balancers. Fargate tasks use the AWSVPC networking mode and provision an Elastic Network Interface (ENI) in your VPC to communicate securely with your resources. With the AWS Command Line Interface (CLI) launching a Fargate task is simple.
aws ecs run-task --launch-type FARGATE --cluster BlogCluster --task-definition blog --network-configuration "awsvpcConfiguration={subnets=[subnet-b563fcd3]}"
It’s also easy to use the console to create task definitions and run tasks with the Fargate launch type.
Once we’ve launched a few tasks we can see them running in our cluster:
You’ll notice that ECS clusters are heterogeneous. They can contain tasks running in Fargate and on EC2.
If we dive a little deeper and look at a task we can see some useful information including the ENI that Fargate provisioned in our VPC and all of the containers used by that task. The logs tab gives me easy access to my CloudWatch Logs for that task as well.
Let’s take a look at the configuration options and pricing details for Fargate.

Pricing

AWS Fargate uses an on-demand pricing model. You pay per per-second for the amount of vCPU and memory resources consumed by your applications. Price per vCPU is $0.0506 per hour and per GB memory is $0.0127 per hour. With Fargate you have 50 configuration options for vCPU and Memory to support a wide range of workloads. The configuration options are below.

CPU (vCPU)Memory Values (GB)
0.250.5, 1, 2
0.51, 2, 3
1Min. 2GB and Max. 8GB, in 1GB increments
2Min. 4GB and Max. 16GB, in 1GB increments
4Min. 8GB and Max. 30GB in 1GB increments

Things To Know

  • You can configure Fargate to closely meet your application’s resource requirements and pay only for resources required by your containers. You can launch tens or tens of thousands of containers in seconds.
  • Fargate tasks run similarly to tasks running on EC2. You can add them to VPCs, configure load balancers, and assign IAM roles.