{"id":2259,"date":"2023-09-20T12:53:55","date_gmt":"2023-09-20T12:53:55","guid":{"rendered":"https:\/\/www.qworqs.com\/?p=2259"},"modified":"2023-12-23T23:28:46","modified_gmt":"2023-12-23T23:28:46","slug":"unprivileged-containers-made-simple-on-debian-12-bookworm","status":"publish","type":"post","link":"https:\/\/www.voodoo.business\/blog\/2023\/09\/20\/unprivileged-containers-made-simple-on-debian-12-bookworm\/","title":{"rendered":"Unprivileged containers made simple on Debian 12 (Bookworm)"},"content":{"rendered":"\n

IMPORTANT NOTE: This is the full version, if you just want to come in, copy some commands, and end up making unprivileged containers under root, THERE IS A SEPARATE POST FOR THAT HERE<\/a><\/strong>.<\/p>\n\n\n\n

0- Intro<\/h1>\n\n\n\n

Don’t let the length fool you, I am trying to make this the simplest and fastest yet most comprehensive tutorial to having LXC (both privileged and unprivileged) up and running on debian bookworm !<\/p>\n\n\n\n

I sent a previous version of this to a friend to spare myself the need to explain to him what to do, and he found the tutorial confusing ! instead of the old arrangement, having colors to denote what lines are for what task, I have decided to SEPARATE THIS INTO PARTS….<\/p>\n\n\n\n

    \n
  1. Intro – About this post (You are already in it)<\/li>\n\n\n\n
  2. LXC info<\/li>\n\n\n\n
  3. Shared system setup (Privileged and unprivileged)<\/li>\n\n\n\n
  4. Privilaged LXC step by step<\/li>\n\n\n\n
  5. Shared setup for unprivileged containers<\/li>\n\n\n\n
  6. Unprivileged LXC run by new user<\/strong>, step by step<\/li>\n\n\n\n
  7. Unprivileged LXC run by root user, step by step<\/li>\n<\/ol>\n\n\n\n

    I hope this clears things up, the color codes will still exist, mostly because I have already done the work ! <\/p>\n\n\n\n

    Why yet another tutorial ?<\/h2>\n\n\n\n

    Most of the tutorials online focus on creating an extra user to use with LXC, that is one way to do it with a few drawbacks, the other way is to create a range of subordinate IDs for the root user, the advantages of this way of doing it are related to “Autostart” and filesystem sharing between host and guest.<\/p>\n\n\n\n

    As per usual, the primary goal of every post on this blog is my own reference, the internet is full of misleading and inaccurate stuff, and when i come back to a similar situation, I don’t want to do the research all over again.<\/p>\n\n\n\n\n\n\n\n

    Part 1: About LXC<\/h1>\n\n\n\n

    Privileged<\/span> VS unprivileged<\/span><\/h2>\n\n\n\n

    Privileged containers are generally unsafe, the only advantage of privileged containers is that is is very easy to setup.<\/p>\n\n\n\n

    Privileged containers share the same root user with the host, so if the container root user gets compromised, the attacker can sneak into the host system, hence, unprivileged is more secure but involves some work initially to setup<\/p>\n\n\n\n

    What is the problem with Privileged containers<\/h2>\n\n\n\n

    It is relatively easy to deploy LXC (Which also happens to be what is powering LXD)… You install it, run a command to create a container, and voila, a whole new Linux system within your host Linux system sharing the same kernel as the host… But there is one caveat, if a malicious user\/application compromises your container, he\/she would have also compromised the host machine automatically, how, the root user on both is the same user !<\/p>\n\n\n\n

    The solution, unprivileged containers<\/h2>\n\n\n\n

    In comes Unprivileged containers, in this setup, we simply either map a User ID to root within the container, or, still use root, but through subordinate IDs<\/strong>, so instead of having the Host’s user id for root (Usually Zero) being also root inside the container, we create a user outside the container (Or a subordinate ID of root), and instruct the kernel to map this user’s ID and treat it as ID zero inside the container, So if a malicious user gets access to the container and ends up breaking out of the container, they will find themselves logged on as a different user, with privileges very close to the privileges of the user nobody, or in other words, barely any privileges<\/p>\n\n\n\n

    Relevant topic: User namespaces<\/h2>\n\n\n\n

    A relevant topic to Unprivileged LXC containers is User namespaces (Starting kernel 3.8), namespaces are created with the functions clone() or unshare().<\/p>\n\n\n\n

    nuff with the theory, What do i need to do ?<\/h2>\n\n\n\n

    You setup LXC, then depending on the type of container and user you need, you may want to setup Linux kernel to use that user as root in the container, but to make that happen, you will need to take a few steps to give that user the required privileges and nothing more than what is required, nothing complicated about those steps either. So let us get started<\/p>\n\n\n\n

    2- Shared system setup<\/span><\/h1>\n\n\n\n

    Before writing this tutorial, I installed a copy of bookworm, enabled SSH, and got to work doing the steps you see below, the steps in this section are the same whether you plan to create privileged or unprivileged containers or both<\/p>\n\n\n\n

    Step 2-1<\/span>: Install everything<\/h3>\n\n\n\n
    apt-get update\napt-get install bridge-utils lxc libvirt-clients libvirt-daemon-system debootstrap qemu-kvm virtinst nmap resolvconf iotop net-tools<\/pre>\n\n\n\n
    Step 2-2<\/span>: Enable IP forwarding<\/h3>\n\n\n\n
    Next, we need to enable IPv4 forwarding by un-commenting a line in sysctl.conf then run sysctl -p, so open sysctl.conf in your favorite linux compatible editor, and uncomment the line<\/p>\n\n\n\n
    net.ipv4.ip_forward=1<\/pre>\n\n\n\n
    Now run the following command for the effects to take place<\/p>\n\n\n\n
    sysctl -p<\/pre>\n\n\n\n
    Step 2-3<\/span>: Host Networking<\/h3>\n\n\n\n
    Before creating any containers, we need to make sure the host can bridge the network to them, in Debian, this is done by editing the file \/etc\/network\/interfaces, there are a few ways to connect the containers, your host can become a DHCP server, or you can connect the containers directly to your router<\/p>\n\n\n\n
    In this setup below, I am connecting the containers directly to the router.. The host machine will have the IP 192.168.7.140, IF YOU ARE USING HYPER-V, YOU WILL NEED TO ENABLE “MAC address spoofing” IN THE HYPER-V VM SETTINGS<\/strong><\/p>\n\n\n\n
    auto br0\n\tiface br0 inet static\n\tbridge_ports eno1\n\tbridge_fd 0\n\taddress 192.168.7.140\n\tnetmask 255.255.255.0\n\tgateway 192.168.7.1\n\tbridge_stp off\n\tbridge_maxwait 0\n\tdns-nameservers 8.8.8.8\n\tdns-nameservers 8.8.4.4\n<\/pre>\n\n\n\n
    3- Privilaged LXC<\/span><\/h1>\n\n\n\n
    To clarify, making a privileged container does not stop you from making unprivileged containers later, BUT, the unprivileged containers need to be different containers \ud83d\ude09 so you might make a privileged one, then replace it with an unprivileged one<\/p>\n\n\n\n
    Step 3-1<\/span>: Download container<\/h3>\n\n\n\n
    The following step is all about downloading your LXC container template ! I chose the mirror with the lowest ping time from me, but you can omit the mirror line altogether<\/p>\n\n\n\n
    MIRROR=http:\/\/ftp.debian.org\/debian lxc-create --name vm142 --template download -- --dist debian --release bookworm --arch amd64<\/pre>\n\n\n\n
    Something unexpected happened while i was doing this, I received an error about a problem downloading, by coincidence, i rebooted<\/strong> the machine and it worked, my theory is that the reboot was irrelevant but if this happens to you, tell me your conclusions in the comments.<\/p>\n\n\n\n
    \"..\/src\/lxc\/lxccontainer.c: create_run_template: 1628 Failed to create container from template\"\n<\/pre>\n\n\n\n
    Right after, you have a brand new LXC container which is unfortunately privileged<\/strong>, you can have it listed with the command “lxc-ls -f” where the f stands for fancy \ud83d\ude09<\/p>\n\n\n\n
    lxc-ls -f<\/pre>\n\n\n\n
    Step 3-2<\/span>: Edit virtual machine config<\/h3>\n\n\n\n
    This container might not be able to start though, some editing of the config file may be necessary !<\/p>\n\n\n\n
    Here is this machines config file, mind the comments, this is meant to be modified to fit your networking setup, so you will need to change the IP address and relevant network address information, the machine name and rootfs path, etc…<\/p>\n\n\n\n
    #this is a modified LXC container config file\n# Template used to create this container: \/usr\/share\/lxc\/templates\/lxc-download\n# Parameters passed to the template: --dist debian --release bookworm --arch amd64\n# For additional config options, please look at lxc.container.conf(5)\n\n# Uncomment the following line to support nesting containers:\n#lxc.include = \/usr\/share\/lxc\/config\/nesting.conf\n# (Be aware this has security implications)\n\n\n# Distribution configuration\nlxc.include = \/usr\/share\/lxc\/config\/common.conf\nlxc.arch = linux64\n\n# Container specific configuration\nlxc.apparmor.profile = generated\n#nesting is for having docker and other similar containerization tech inside the container, dissable it if you don't want such virtual machines in the virtual machine\nlxc.apparmor.allow_nesting = 1\nlxc.rootfs.path = dir:\/var\/lib\/lxc\/vm142\/rootfs\nlxc.uts.name = vm142\n\n# Initial Network configuration, disabled...\n#lxc.net.0.type = veth\n#lxc.net.0.link = lxcbr0\n#lxc.net.0.flags = up\n\n#the above config was dissabled, so net.0 altogether is better left empty\nlxc.net.0.type = empty\n\n\n#Now, add networking\n\nlxc.net.1.type = veth\nlxc.net.1.flags = up\nlxc.net.1.link = br0\nlxc.net.1.name = eth0\nlxc.net.1.ipv4.address = 192.168.7.142\/24\nlxc.net.1.ipv4.gateway = 192.168.7.1\n\n\n#App armor profile for this PRIVILEGED container\nlxc.apparmor.profile = generated\n\n\n#If you want this container to start with the host, uncomment the following\n#lxc.start.auto = 1\n#lxc-start.delay = 10\n# #the order, the higher the earlier ;) \n#lxc.start.order = 30\n\n\n# Container specific configuration (Not initially there)\nlxc.tty.max = 4\nlxc.pty.max = 1024\n\n<\/pre>\n\n\n\n
    Problem : One remaining problem was that the virtual machine was getting 2 IP addresses, one static that we set above, and one dynamic via DHCP, turns out the \/etc\/systemd\/network\/eth0.network forced the machine to get DHCP, so i went in and commented all the lines inside that file !<\/p>\n\n\n\n
    Another problem that came up was DNS resolution, the file you need to edit is sftp:\/\/192.168.7.123\/etc\/systemd\/resolved.conf, simply add the following two lines<\/p>\n\n\n\n
    DNS=8.8.8.8\nFallbackDNS=8.8.4.4<\/pre>\n\n\n\n
    Step 3-3<\/span>: Start the machine and change credentials<\/h3>\n\n\n\n
    Now, after starting the machine, you will need to login to it, to start the virtual machine and do that, issue the command<\/p>\n\n\n\n
    lxc-start -n vm142 -d\nlxc-attach -n vm142<\/pre>\n\n\n\n
    Now, you can use the passwd<\/strong> command to change the container’s password, and you would probably want to install “apt-get install ssh openssh-server”, this way you can login to it with putty or any other SSH client<\/p>\n\n\n\n
    4- Unprivileged LXC containers (Both)<\/h1>\n\n\n\n
    Whatever in this section applies to unprivileged containers, whether root user or any other user<\/p>\n\n\n\n
    Step 4-1<\/span>: Enable Unprivileged User Namespaces <\/h3>\n\n\n\n
    it is enabled by default, To make sure that it is, run the command below, if it returns “kernel.unprivileged_userns_clone = 1<\/strong>” you are good to go.<\/p>\n\n\n\n
    sysctl kernel.unprivileged_userns_clone<\/pre>\n\n\n\n
    if for any reason it is not enabled (0), you can enable it by adding it to \/etc\/sysctl.d<\/strong>…. by editing the file “\/etc\/sysctl.d\/00-local-userns.conf” and adding the following line, if the file does not exist, create it<\/p>\n\n\n\n
    kernel.unprivileged_userns_clone=1<\/pre>\n\n\n\n
    Once done, run the command<\/p>\n\n\n\n
    service procps restart<\/pre>\n\n\n\n
    5- Unprivileged container under new user<\/h1>\n\n\n\n
    Step 5-1<\/span>: Create the user<\/h3>\n\n\n\n
    You can call the user whatever you want, I chose to call the user lxcadmin<\/strong>, this is an arbitrary <\/strong>choice, To create a user we issue the following command.<\/p>\n\n\n\n
    adduser lxcadmin<\/pre>\n\n\n\n
    The output of the adduser command should be something like<\/p>\n\n\n\n
    Adding user `lxcadmin' ...\nAdding new group `lxcadmin' (1001) ...\nAdding new user `lxcadmin' (1001) with group `lxcadmin (1001)' ...\nCreating home directory `\/home\/lxcadmin' ...\nCopying files from `\/etc\/skel' ...\n...\nAdding new user `lxcadmin' to supplemental \/ extra groups `users' ...\nAdding user `lxcadmin' to group `users' ...<\/pre>\n\n\n\n
    So here, our user gets the ID 1001 (Since i already have a user with the ID 1000 and the root user with the ID 0. Now if we inspect the 2 files \/etc\/subuid (The subordinate uid file) and \/etc\/subgid, we will find the following content in both (Identical contents in files).<\/p>\n\n\n\n
    yazeed:100000:65536\nlxcadmin:165536:65536<\/pre>\n\n\n\n
    What the above means is that user lxcadmin has a range of UIDs starting with 165536<\/strong> and has 65536 extra <\/s>UIDs total, so the last UID that lxcadmin can use is 165536 + 65536 – 1 = 321071<\/strong>, and the next user we add will start at 321072.<\/p>\n\n\n\n
    So to recap this user has a subordinate ID range from 165537 TO 321071<\/strong>, notice i added one to the starting number since the first number is not a subordinate ID, but rather the user’s default ID.<\/p>\n\n\n\n
    Step 5-2<\/span>: Network adapter quota<\/h3>\n\n\n\n
    New users generally do not have the ability to add a container to a bridge, for that you will need to give the user a network device quota, this quota is defined in the file \/etc\/lxc\/lxc-usernet<\/strong>, the initial quota for unprivileged users is zero, so edit the file and add the following lines, depending on what adapters you would like to allow lxcadmin to connect containers to, the format is user<\/strong> type<\/strong> bridge<\/strong> quota<\/strong><\/p>\n\n\n\n
    lxcadmin veth lxcbr0 10\nlxcadmin veth br0 10<\/pre>\n\n\n\n
    Notice that you can<\/strong> replace the user with a group name, but that is a subject of a different post…<\/p>\n\n\n\n
    Now you will need to copy the file \/etc\/lxc\/default.conf<\/strong> to the user’s home directory, in my case under \/home\/lxcadmin\/.config\/lxc\/default.conf<\/strong>, if the config directory does not exist, create it, now edit this file you just created and depending on the user you are using (I am using the second user, hence the numbers, yours will differ unless your user is the second one added, copy the values from \/etc\/subuid)\u2026<\/p>\n\n\n\n
        lxc.idmap = u 0 165536 65536\n    lxc.idmap = g 0 165536 65536\n<\/pre>\n\n\n\n
    Now, we are closer than ever to making it run, we need to create our first container, unlike the privileged “lxc-create mycontainer” this one is slightly more complicated (The solution is below to make things unprivileged and secure again)<\/p>\n\n\n\n
    systemd-run --unit=my-unit --user --scope -p \"Delegate=yes\" -- lxc-create -t download -n my-container<\/s>\nlxc-create -t download -n myunprivcontainer -- -d debian -r bookworm -a amd64<\/pre>\n\n\n\n
    Don’t expect this to work yet…. the following contgainer config file was automatically created<\/p>\n\n\n\n
    # Template used to create this container: \/usr\/share\/lxc\/templates\/lxc-download\n# Parameters passed to the template: -d debian -r bookworm -a amd64\n# For additional config options, please look at lxc.container.conf(5)\n\n# Uncomment the following line to support nesting containers:\n#lxc.include = \/usr\/share\/lxc\/config\/nesting.conf\n# (Be aware this has security implications)\n\n\n# Distribution configuration\nlxc.include = \/usr\/share\/lxc\/config\/common.conf\nlxc.include = \/usr\/share\/lxc\/config\/userns.conf\nlxc.arch = linux64\n\n# Container specific configuration\nlxc.apparmor.profile = generated\nlxc.apparmor.allow_nesting = 1\nlxc.idmap = u 0 165536 65536\nlxc.idmap = g 0 165536 65536\nlxc.rootfs.path = dir:\/home\/lxcadmin\/.local\/share\/lxc\/myunprivcontainer\/rootfs\nlxc.uts.name = myunprivcontainer\n\n# Network configuration\nlxc.net.0.type = veth\nlxc.net.0.link = br0\nlxc.net.0.flags = up\n<\/pre>\n\n\n\n
    libpam-cgfs is already installed (It was a dependancy in the apt-get install above), libpam-cgfs is a Pluggable Authentication Module (PAM) to provide logged-in users with a set of cgroups which they can administer. This allows for instance unprivileged containers, and session management using cgroup process tracking.<\/p>\n\n\n\n
    Configure AppArmor<\/h3>\n\n\n\n
    App Armor is enabled on Debian 10 (buster) and after by default, AppArmor is recommended as it adds a layer of security which may prove vital for a system running your virtual machines.<\/p>\n\n\n\n
    to check whether it is enabled on your system or not, you can run the following command<\/p>\n\n\n\n
    cat \/sys\/module\/apparmor\/parameters\/enabled<\/pre>\n\n\n\n
    If the above returns the letter Y, AppArmor is enabled, and you need to set it up to allow for our unprivileged setup<\/p>\n\n\n\n
    6- Unprivileged container under root subordinates<\/h1>\n\n\n\n
    This is the most interesting setups, It is a no compromise setup where you can have a container run with all the features you see in privileged containers, while still maintaining the security provided by the unprivileged setup above (More or less)<\/p>\n\n\n\n
    Step 6-1<\/span>: root Subordinates:<\/h3>\n\n\n\n
    the first step is to allocate a uid and gid range to root user in \/etc\/subuid<\/strong> and \/etc\/subgid<\/strong>. This is because the root user, unlike users added with adduser, does not have subordinate IDs by default, so in short, figure out what the next range of IDs is, and assign them to root by adding a line similar to the following at the top of the list in those 2 files, In my case, lxcadmin has the last range, 165536:65536 means the next id is (165536 + 65536 = 231072<\/strong>), And i would like a million subordinate IDs so i can hand every machine a different set of IDs which should increase security even farther.<\/p>\n\n\n\n
    root:231072:1000000<\/pre>\n\n\n\n
    adduser will recognize the new range when you use it next time, and start from there<\/p>\n\n\n\n
    And reflect that range in \/etc\/lxc\/default.conf using lxc.idmap entries similar to those above.<\/p>\n\n\n\n
    root does not need network devices quotas and uses the global configuration file, so those steps from the above are not needed.<\/p>\n\n\n\n
    Any container you create as root from that point on will be running unprivileged, able to auto-start, and share filesystems !<\/p>\n","protected":false},"excerpt":{"rendered":"
    IMPORTANT NOTE: This is the full version, if you just want to come in, copy some commands, and end up making unprivileged containers under root, THERE IS A SEPARATE POST FOR THAT HERE. 0- Intro Don’t let the length fool you, I am trying to make this the simplest and fastest yet most comprehensive tutorial […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4,173,139],"tags":[],"class_list":["post-2259","post","type-post","status-publish","format-standard","hentry","category-linux","category-lxc","category-virtualization-linux"],"_links":{"self":[{"href":"https:\/\/www.voodoo.business\/blog\/wp-json\/wp\/v2\/posts\/2259","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.voodoo.business\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.voodoo.business\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.voodoo.business\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.voodoo.business\/blog\/wp-json\/wp\/v2\/comments?post=2259"}],"version-history":[{"count":137,"href":"https:\/\/www.voodoo.business\/blog\/wp-json\/wp\/v2\/posts\/2259\/revisions"}],"predecessor-version":[{"id":3139,"href":"https:\/\/www.voodoo.business\/blog\/wp-json\/wp\/v2\/posts\/2259\/revisions\/3139"}],"wp:attachment":[{"href":"https:\/\/www.voodoo.business\/blog\/wp-json\/wp\/v2\/media?parent=2259"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.voodoo.business\/blog\/wp-json\/wp\/v2\/categories?post=2259"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.voodoo.business\/blog\/wp-json\/wp\/v2\/tags?post=2259"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}