QWORQS – A blog by Yazeed Almadaeen
What services are listening on what ports ?
netstat -tunlp
or the all new ss which shares the same command options
ss -tunlpfor a better view of the files ! (Depending on your setup, you might be able to know which php script is generating outgoing traffic)
lsof -nP -iTCP -sTCP:LISTENSo, before we start, I assume you have already installed Varnish the casual way, and that you have made sure you do not have something else occupying port 80, if nginx for example is listening on port 80, try this post to switch it to a different port (Changing the default port 80 on nginx), if it is a different app, just follow that app’s instructions to change it’s port before asking varnish to listen on it.
To check that port 80 is not occupied and free to use, try the following command
netstat -anpe | grep "80" | grep "LISTEN" If for example you want to know what ports nginx is listening to, try one of the following commands netstat -tlpn| grep nginx or ss -tlpn| grep nginx
Within the results, check if any are using port 80, mind you, a service using port 8083 for example will show up, you need to see if anything is using port 80 , Now, install varnish with the following command.
apt-get install varnish
As you may have noticed, and probably the reason why you are here, varnish will not work !
this is somewhat of an old problem, since Debian moved to systemD back with the Debian 8 release (Jessie), instead of editing the file in /etc/default/varnish, you will need to create a file in /etc/systemd/system/ and name it varnish.service, the contents of such a file should look like the following, note that xxx.xxx.xxx.xxx is the IP varnish should be listening on, one of the IPs assigned to the machine running varnish.
So to run the following command
systemctl edit varnish.service
[Unit] Description=Varnish HTTP accelerator Documentation=https://www.varnish-cache.org/docs/6.1/ man:varnishd [Service] Type=simple LimitNOFILE=131072 LimitMEMLOCK=82000 ExecStart=/usr/sbin/varnishd -j unix,user=vcache -F -a xxx.xxx.xxx.xxx:80 -T localhost:6082 -f /etc/varnish/default.vcl -S /etc/varnish/secret -s malloc,256m ExecReload=/usr/share/varnish/varnishreload ProtectSystem=full ProtectHome=true PrivateTmp=true PrivateDevices=true [Install] WantedBy=multi-user.target
Once you have added the file execute the following
systemctl daemon-reload systemctl restart varnish
Note that the configuration file above uses RAM to cache the content, My recommendation is to use DISK (Disk is cached in ram in a more dynamic and more useful way utilizing all the ram you are not using, while keeping it available to any app that needs it), but that is just me…
To switch from RAM to File system, replace the following in the above as follows
"-s malloc,512m" becomes "-s file,/path/to/cahce/file.bin,100G"
Now, if you want to flush the varnish cache, you don’t have to remove everything ! you can use the varnishadm command to control the cache
Don’t let the keyword “Ban” fool you, it bans the existing copy of the cache, but does not prevent it from getting re-cached
- A domain's cache: varnishadm "ban req.http.host ~ www.example.com"
- File type: varnishadm "ban req.url ~ .css"
- Both the above: varnishadm "ban req.http.host ~ www.example.com" && req.url ~ .css"
- a URL varnishadm "ban req.url ~ /directory/andmaybefile"
- whole cache: varnishadm "ban req.url ~ ."
- Every png on domain: varnishadm ban req.http.host == example.com '&&' req.url '~' '\\.png$'
First of all, after using yt-dlp to download 2 webm files for a video, you can combine them without reencoding using the following command (See here), Just make sure both files are VP8 or VP9
ffmpeg -i ao.webm -i vo.webm -c:v copy -c:a copy output.webm
To extract the section from the file resulting from the first command above
1- From second x, and duration in seconds
ffmpeg -ss 285 -i g_in.mkv -t 5475 -map 0 -c copy g_out.mkv
2- From second x to second y
ffmpeg -copyts -ss 4633 -i g_in.mkv -to 5470 -map 0 -c copy g_p2.mkv
Converting a file you have downloaded using 1 or any other file into MP4 (H264), since some windows computers will not play a webm file !
ffmpeg -i source264.mp4 -c:v libx265 -crf 28 -preset fast -c:a aac -b:a 128k -filter:v fps=25 out265.mp4
If you want to cut a part of the video, without re-encoding it
ffmpeg -i input.mp4 -vcodec copy -acodec copy -ss 00:01:00.000 -t 00:00:02.000 output.mp4
Nvidia graphics card to convert H265 (MKV) to H264 MP4, Using nvidia hardware encoder to encode video into H264 or H265 !
To checks whether hardware encoders are available or not, run the command
ffmpeg -encoders | findstr /ic:"NVIDIA"
If the following two lines are in the command, you can use the nVidia encoder, the first codec is H264, and the second is for H265 (HEVC)
V....D h264_nvenc NVIDIA NVENC H.264 encoder (codec h264) V....D hevc_nvenc NVIDIA NVENC hevc encoder (codec hevc)
ffmpeg.exe -vsync 0 -hwaccel cuda -i <input_file> -map 0 -c:a copy -c:v h264_nvenc -pix_fmt yuv420p -preset hq <output_file>
-vsync : Synchronize video audio and metadata using the video timestamp
-hwaccel cuda : Use nVidia’s cuda for hardware accelleration
Example, Convert mkv to mp4 (Tested OK)
Now, with the above out of the way, the following command should encode your 1080P mkv H265 video to H264 ! all within GPU, so this re-encodes an nVidia compatible format to another nVidia compatible format, on my 1650 card, it was encoding at 12x. this provides better compatibility, if smaller file size and quality are what you seek, then you should do it the other way around
ffmpeg -i "1080p.mkv" -c:v h264_nvenc -pix_fmt yuv420p -minrate 500k -maxrate 1000k -c:a mp3 -b:a 128k "1080p.mp4"
The other way around
ffmpeg -i "1080p.mp4" -c:v hevc_nvenc -pix_fmt yuv420p -minrate 200k -maxrate 1000k -c:a aac -b:a 128k "1080p.mkv"
NOTE: The above does everything within the GPU, if for example you wanted the decoding on CPU, that will make things much slower because the decoded video (Huge) will still need to be copied to the GPU,
Encoding H265 with my entry level GPU (1650) is around five to fifteen times faster than with CPU (6 core, 12 thread i7-4930K), CPU was running at around 50% load through the encoding
Now one very common task people want to execute is converting their old, bulky DVD collection to H265 (Or H264 if they value compatibility over size and clarity), DVD files are usually on a DVD in the Video_TS folder, and the AudioTS folder, So this will create a few cases
Case 1: Audio_ts folder is empty, Video TS folder has files that you know the order they should be displayed in, the objective is to put them all in one video file (Assuming MKV but the container is your choice), in this case, I usually start by converting all the videos to H265, then combine them, here, most of my videos are interlaced (that will be dealt with with yadif), and sometimes, they are files of different resolutions, so I will unify their size
ffmpeg -i "v1.VOB" -c:v hevc_nvenc -c:a aac -b:a 256k -vf yadif,scale=1920:1080 -x265-params "crf=22:min-keyint=25:keyint=50" -preset slow "d1.mkv"
Now, create a list of the files
(for %i in (*.mkv) do @echo file '%i') > mylist.txt
And concatenate the videos
ffmpeg -f concat -i mylist.txt -c copy output.mkv
Now, to batch process a folder on the command line... in this example i am lowering the resolution of files to FHD from 4K
for /f "tokens=1 delims=." %a in ('dir /B *.mp4') do ffmpeg -i "%a.mp4" "%a.1080.mp4"for /f "tokens=1 delims=." %a in ('dir /B *.mp4') do ffmpeg -i "%a.mp4" -vn -acodec copy "%a.aac"
for i in *.mp4; do ffmpeg -i "$i" -vn -acodec copy "${i%.*}.aac"; done
Split audio file
ffmpeg -i 2024-07-27_20_38_2.m4a -f segment -segment_time 1740 output_%03d.m4aHere is a case where my MKV file contains the audio in AAC, and the video in H264, As you slide the slider in VLC media player, it tells you what the section is about, i should be able to copy the streams exactly as they are, but I was concerned that the “Slider Hints” would disappear, hence I split it into audio and video without re-encoding, checked that the hints are still there in the video, then combined them into an MP4, also without encoding, the reason I wanted the container to be MP4 is that sometimes I access my network files in a browser, and browsers seem more comfortable playing MP4 files than MKV
* ffmpeg -i My_Dinner_With_Andre_1981.mkv -vn -acodec copy ao.aac
* ffmpeg -i My_Dinner_With_Andre_1981.mkv -an -c:v copy vo.mp4
* ffmpeg -i ao.aac -i vo.mp4 -c:v copy -c:a copy My_Dinner_With_Andre_1981.mp4
3D printing
When i 3D print a part, I usually want it to be 10X faster, with good compression and no audio, I also want the idle time at the end of the video cropped, here is the command to do tat
ffmpeg -i 20240927_145610.mp4 \
-t 00:12:24 \
-vf "select='not(mod(n,5))',setpts=N/FRAME_RATE/TB,scale=1280:-1" \
-an -c:v libx265 -crf 30 -preset slow 20240927_145610_5x.mp4
-t 00:12:24 → stops processing after 1 hour 2 minutes of input (62 minutes/5).
select=’not(mod(n,10))’ → keep 1 out of every 10 frames → 10× speedup.
setpts=N/FRAME_RATE/TB → adjust timestamps so playback is fast, not choppy in duration.
scale=1280:-1 → shrink to 1280 px width, preserving aspect ratio.
-an → remove audio.
-c:v libx265 → H.265 video codec (much smaller than H.264 for the same quality).
-crf 30 → aggressive compression (higher = smaller file, lower quality).
-preset slow → better compression efficiency (slower encode).
Reload
firewall-cmd --reloadDealing with zones
* firewall-cmd --get-zones <- Show all zone names * firewall-cmd --list-all-zones <- Detailed info about all zones * firewall-cmd --zone=zone-name --list-all <- show everything about a certain zone * firewall-cmd --get-default-zone <- What is the current default zone * firewall-cmd --set-default-zone zone-name <- set new default zone * firewall-cmd --get-active-zones <- List the active zones and the interfaces assigned to them * firewall-cmd --permanent --zone=public --change-interface=enp1s0 <- connect the public zone to eth0, REQUIERS RELOAD
In this mini tutorial, I will be adding the user kareem to the system, and allow kareem to sftp into a web directory where he can post his web design work, as usual, the steps first, then whatever explanations !
There are two ways to do this, one to add one user, the other to add a group of users, you can either pick one, or do both !
The part in common between both solutions
apt-get install openssh-server adduser kareem Then enter a new password twice for kareem
The interesting thing about this sftp user business is that the directory we will specify as the root for the user kareem has to be owned by root ! so go ahead and create the directory /var/www/html/usr/kareem, then execute the following commands
chown root:root /var/www/html/usr chmod 755 /var/www/html/usr chown kareem:kareem /var/www/html/usr/kareem
Now, the user kareem owns a directory within his root directory that he can write to, and can not write outside that directory since he does not have the OS permissions, Now, let us add kareem to the list of people who have sftp access but not ssh access.
Edit /etc/ssh/sshd_config and append the following to the document
Match User kareem ForceCommand internal-sftp PasswordAuthentication yes ChrootDirectory /var/www/html/usr PermitTunnel no AllowAgentForwarding no AllowTcpForwarding no X11Forwarding no
Now, restart the service by executing the following command
systemctl restart ssh
You are done, try connecting with something like winSCP
Besides winSCP, you can also simply mount the linux filesystem where you have permissions on your windows machine, here are the complete instructions on how to do that https://www.qworqs.com/2022/10/09/mounting-a-remote-linux-file-system-as-a-windows-drive/
I haven’t played any games in 20 years, up until early this month, I played a simple game on my phone, nice, but I’m not addicted, no urge to pick up the phone
So, having no windows PC of my own, i decided to install the open source supertuxkart on my debian (gnome) machine
the game is a joke compared to modern day graphics and physics, but it is fun, with things like bombs and banana skins and stuff, i don’t think i will be playing it often, but there you have it
The answer to this should be simple, I initiated the test with
badblocks -nsv /dev/sdb
, first, interrupt bad blocks with ctrl+c, the output should be
Checking for bad blocks in non-destructive read-write mode From block 0 to 1953514583 Checking for bad blocks (non-destructive read-write test) Testing with random pattern: 0.92% done, 49:38 elapsed. (0/0/0 errors) 21.32% done, 18:49:24 elapsed. (0/0/0 errors) Interrupted at block 416437376 Interrupt caught, cleaning up
Okay, so we know what blocks it was supposed to check (1 through 1953514583), and where it was interrupted (416437376)
So i will ask it to resume testing from where it finished (-1), up to the end
badblocks -nsv /dev/sdb 1953514583 416437375 n = Non destructive s = Show progress v = tell us about what you find !
The new run should tell you the percentage correctly, but the time counter will be reset to zero, as it is only counting how long this run has been running for
One thing to note is that bad blocks can be used to instruct the filesystem to avoid the bad blocks, but it also allows the disk’s firmware to substitute bad blocks with spare blocks, so that the disk works again with no intervention from your end !
So for my 2TB hard drive…
416437375 = 21% (13 hours) 619014719 = 31.6% (+23:22) 627995199 = 32.15% (+1:04) 667782398 = 34.18% (+4:46) 715469885 = 36.62% (+5:44) 827834875 = 42.38%
While running the tests, you might want to keep an eye on the hard drive temperature with a command like
hddtemp /dev/sdb
To create a log file of the bad blocks, every run should have it’s own file !
badblocks -nsv -o /root/badblocks3.txt /dev/sdb 1953514583 627995198
The concatenation of those files you are creating is very useful in creating a file system if you ever decide to format the drive later !, but the recommended way is using badblocks with the other disk tools directly
while the test is running, you will see 3 numbers that correspond to readerror/writeerror/corruptionerror
To begin with, the links here are Amazon associate links, what this means is that buying things using those links will result in a commission for me ! to recap. “As an Amazon Associate I earn from qualifying purchases.”
What I am trying to do is to create a cover that allows me to switch between USB OTG devices easily at the workshop, so i created a 3D printed connector that fits with the following parts, the STL for that model will be provided once I use it and make sure it works, In addition to the pixel 6 pro, have made the 3D printed connector holder for the Samsung Galaxy S10, S10+, Ulefone Power 5, Ulefone Armor 3,
The external camera also has a 3D printed case design which I will also share… the screen protector has nothing to do with the project, but i am getting it anyways because it is nice if you have screen protection at the workshop, the screen protection uses UV curing adhesive ! which may or may not ruin your oleophobic coating on your screen (remains to be seen)
Screen protector Here said to work best with the fingerprint sensor (Once you use the penny trick), and tough enough to stay put with those curved edges !
2 phone covers here (Shaded Spurce (Green) and Slate (Grey))
4 of USB C Magnetic Adapter here, 2 Elbow and 2 Flat Elbow
Google’s Air Buds (Pixel) are on sale for $99 (here) and the pro are discounted at $199 (Here), I personally like my ugreen with qualcom chipset, So i don’t think i will get the google ones any time soon, but they are here for ref.
A couple of hours ago, i received a video that is 50 frames per second, and compressed in H264, the video was 58MB, and she wanted it less than 15 to send it via email, the video was 1:45 long, so i re-encoded it in H-265 but she had a problem playing it (No codec), so i decided to re-encode it with VP9 (webm).
to arrive at a number less than 10, i needed to be encoding at around 1 MegaBIT per second, now, to do this, I made a 2pass encoding with ffmpeg as follows
ffmpeg -i source.mp4 -c:v libvpx-vp9 -b:v 1M -filter:v fps=25 -pass 1 -an -f null /dev/null && \ ffmpeg -i source.mp4 -c:v libvpx-vp9 -b:v 1M -filter:v fps=25 -pass 2 -c:a libopus out.webm
The first pass collects statistics about the source video in a text log file, the second pass encodes the new video, from the options above, i have taken the frame rate to 25fps (from 50), and instead of defining the crf, i simply told ffmpeg what the biterate I need is, which is 1Mbit per second (Every 8 seconds, 1 MBYTE)
The previous one, H-265 was done with the command
ffmpeg -i source264.mp4 -c:v libx265 -crf 28 -preset fast -c:a aac -b:a 128k -filter:v fps=25 out265.mp4
the H265 was smaller due to the crf factor used, as well as the lower frame rate
Like the name implies, this is a cheat sheet to quickly find the command you need, they are ordered by the frequency a command is used, or at least what i think is going to be needed more frequently, I have also grouped them by function
The container name in the examples is mycontainer, it is just a name that you will need to replace with your own container name, the container ID here is always 12345abcdef
| Command | Arguments | What it does |
| ============> | Containers – list | |
| docker container ls | Display running containers | |
| docker container ls -a | a: also show containers that are not running | Display all containers, running or not |
| docker ps | Show running containers | PS is the same as LS but older |
| ============> | Containers – Run | |
docker run --name mycontainer -i -t imagename | 1- The name of the container to run (mycontainer)2- The i flag indicating you’d like to open an interactive SSH session to the container. The i flag does not close the SSH session even if the container is not attached.3- The t flag allocates a pseudo-TTY which much be used to run commands interactively.4- The base image to create the container from ( imagename). | Runs the container, and leaves you on a shell prompt that executes commands on that container (As if you have ssh-ed into it) |
docker run --name | -d for running the container in the background | |
docker stop my_container | Stop the running container | |
docker exec -it | -it flag allows you to run a container in interactive mode | If this doesn’t work, you may not have bash installed, you can try the next command Gives you access to the shell, much like opening an SSH session to the container |
docker exec -it username/ | ||
| ctrl+p followed by ctrl+q | Detach from container |
Sometimes, accessing a container throght the command line may not be enough, there is a chance you want to access it for file transfer for example, in that case, you want port 22 exposed, and you want to be connected to it like you would connect to a virtual machine