Author: Yazeed

ethtool -K eno1 gso off gro off tso off

To check the state

ethtool –k eno1 | grep tcp

I am not sure this requires any introduction !

While the ChatGPT app can be accessed through, the API portal is at platform.openai.com, and even though the website provides a free tier, the API does not, you must have prepaid credits in order to use it, the minimum is $5, those $5 are pay as you go, you use them up by sending more API calls.

GPT 4o is rumored to be 10 Trillion parameters !! for comparison, deepseek, which is an amazing LLM that came out lately is 671 Billion parameters, meaning it is 7% of the size of ChatGPT !

Models and API pricing

If you are familiar with python, you probably are also familiar with pip, and very likely familiar with venv (Virtual environments)

There is nothing special in particular about this for AI, it is exactly as you would for any other installation.

For AI, i would totally recommend anaconda, but if for some reason that is not an option, this option will do 100% of the time

So, you need to start by installing Python 3 !, On debian, I would just use the repository with apt, it may be true at the time of writing that in the repo it uses 3.11 rather than the latest 3.13, but that is absolutely fine

sudo apt update
// The following command should do
sudo apt install python3
//But i would much rather install everything python in one go
apt install build-essential wget git python3-pip python3-dev python3-venv \
     python3-wheel libfreetype6-dev libxml2-dev libzip-dev libsasl2-dev \
     python3-setuptools

Now, with that out of the way, navigate to the project’s folder (Assuming you have downloaded a project for example), and create a virtual environment

python3 -m venv venv

Now, you can activate that project with

source venv/bin/activate
//On windows, the above should look something like
venv\Scripts\activate

That is basically it, you will now need to, from within the command prompt of the venv, install dependencies either one by one using the pip command, or point it to a file containing the dependencies, for example

pip install -r requirements.txt

You should now be good to go !

Google colab is a hosted Jupyter Notebooks service from google

It appears to me as though the name is also a pun, a short notation for the word collaboration, and also a CO-Lab as in Jupyter lab

Launch

Within the context of our AI tutorial, start by activating your conda environment (Or venv)

Once active, run the following command to launch Jupyter

jupyter lab

Jupyter Notebook = The original three languages that were supported were Julia, Python and R. Sure there is a missing E in the words, but i guess the name is 90% cool. The logo pays homage to Galileo’s discovery of the moons of Jupiter, So I am not sure whether the name can be called a pun since the one of them is a shortening of three words and not a name, and a name with a misspelled letter.

Enough about the name and logo, let us get into what a Jupyter Notebook is

Technically, it is a JSON file with a ipynb extension, in reality, it is a way to create a web document “With live code”, equations, among other things

the types of cells in a notebook are

• Code Cells: Execute code and display the output.
• Markdown Cells: Write formatted text using Markdown.
• Raw NBConvert Cells: Include content that is not evaluated by the notebook kernel

So, the markup inside that JSON file is not specific to AI per say, it is used in many science fields, but as you work your way through this blog, you will see how important it is for what we are trying to achieve ! important in the sense that it makes development simpler, it is not a thing you will use in your final product

In any case, to start Jupyter Notebook. activate your Conda environment, then run the following command from the terminal where the current directory is your project directory !

jupyter lab

Then, assuming you are working on a project that your downloaded from github for example, you can open the ipynb files found in there by clicking on them form the menu on the left !

Code cells get executed by a background python thread (Kernel) in the background, step by step when you go to that cell and click shift+enter

What is Anaconda ?

Conda, Like pip, is a python package manager, but conda is probably undisputed as the more thorough solution of the two, with better support for non-python packages (pip has very limited support) and support for more complex dependency trees

To clarify things, conda is the package manager while Anaconda is a bigger bundle, if you want to install conda alone, you are probably looking to install Miniconda. Anaconda is a set of about a hundred packages including conda, numpy, scipy, ipython notebook, and so on.

So, let us go through installing and using Anaconda on all 3 platforms, Windows, Linux and Mac

Linux

On Debian, there is no Anaconda package, to install, you will need to download the download script from anaconda and install it (Or conda, or miniconda for that matter) , you can add miniconda to apt using the “https://repo.anaconda.com” repo if you are willing to add it (apt install conda), but here I will assume you will just install Anaconda, and the only orthodox way to do that is with the installation script

Download the Anaconda installer from the Anaconda Website (There is a button that reads skip registration if you don’t want to give them your email address)

https://www.anaconda.com/download

Navigate to the downloads folder, and execute the script just downloaded, in my case, the script’s name was Anaconda3-2024.10-1-Linux-x86_64.sh so I execute the following

cd /home/qworqs/Downloads
chmod 0777 Anaconda3-2024.10-1-Linux-x86_64.sh
./Anaconda3-2024.10-1-Linux-x86_64.sh

After accepting the agreement, I see a message asking me to accept the license agreement, hit enter, take a look at the license agreement, hit the letter (q) to exit the agreement, then you will be asked if you accept the agreement, assuming you agreed to it, you will next be presented with….

Anaconda3 will now be installed into this location:
/home/qworqs/anaconda3

To which i accepted the suggested location

Now, I opt to keep the installer in the downloads directory just in case something goes wrong, but you can safely delete the 1GB installer if you like !

At the end of the installation, the installer offers to update your shell, in my case, i opted NOT TO, if you opted otherwise, you can always “set auto_activate_base” to false….

Do you wish to update your shell profile to automatically initialize conda?
This will activate conda on startup and change the command prompt when activated.
If you'd prefer that conda's base environment not be activated on startup,
   run the following command when conda is activated:

conda config --set auto_activate_base false

You can undo this by running `conda init --reverse $SHELL`? [yes|no]

Once i answered no, i was presented with the following message

You have chosen to not have conda modify your shell scripts at all.
To activate conda's base environment in your current shell session:

eval "$(/home/voodoo/anaconda3/bin/conda shell.YOUR_SHELL_NAME hook)" 

To install conda's shell functions for easier access, first activate, then:

conda init

Thank you for installing Anaconda3!

The environment

For convinience, let us start by adding our Anaconda3 installation to system path

//First, add anaconda to path by editing either ./bashrc OR ~/.bash_profile (Depending on which one you have), and adding the following to the bottom of the file

export PATH=~/anaconda3/bin:$PATH

Now, to apply the changes, you should either close the terminal window and re-open it, or run the command “source ~/.bashrc” or “source ~/.bash_profile”

To check whether the magic happened, run the command “conda –version“, in my case, that returned “conda 24.9.2”

Now, from this stage on, conda is installed, but to be able to use it, you should have a project ! so now you can move on (The index page), and I will explain how to run your project where needed, here, for completion, I will assume you have a project and put the instructions here (You know you are in the project dir when you see a yaml file commonly called environment.yml)

So, again, to activate an environment, there is a yaml file for every project that contains the dependencies of that project ! let us assume you are in your project’s directory, and the yaml file is called environment.yml , the following command will create a python sub-environment, and install all the dependencies in that yaml file, be sure you are in the directory with the yaml file

Now, to create a virtual environment, cd into the directory that has your project and run the following

conda env create -f environment.yml

Once the above is done downloading and installing, you should get a message like the one below

#
# To activate this environment, use
#
#     $ conda activate projectName
#
# To deactivate an active environment, use
#
#     $ conda deactivate

Now, next time, when you open a terminal, and want to activate an environment

1- conda init (conda deactivate to reverse), you only need to do this once
2- open a new shell
3- conda activate ProjectName (Also, conda deactivate to exit)

Whisper is the leading audio transcription service that can be installed on your own computer, my personal experience with whisper is excellent, in fact it is the best one i know.

• The official software, Github =>
• openAI Whisper implementation in C++ for CPU or GPU ! =>
• OpenAI Faster Whisper: Here Like the one above, but 5X faster ! CPU and GPU

What a pleasant surprise this is, something you can run locally on your computer, or use it for a fraction of the cost that comes with OpenAI or Anthropic’s Calude !

DeepSeek-V3 is completely open-source and free. (https://github.com/deepseek-ai/DeepSeek-V3)

If you don’t have the hardware resources for it, it is also available through a website identical to that of ChatGPT and an incredibly affordable API.

How affordable ?

Deep Seek: $0.14 per million input tokens and $0.28 per million output tokens.
Claude AI : $3.00 per million input tokens and $15.00 per million output tokens
ChatGPT : $2.50 per million input tokens and $10.00 per million output tokens

So, the bottom line is that deep seek is fifty times cheaper than Claude AI, and around 35 times cheaper than openAI ! that is, two percent and three percent of the price, But what about quality

in most scenarios, it is comparable, in some cases Deep Seek wins, in other cases, claude or ChatGPT wins, but it is up there obviously !

1- Installing

1.1 – Linux

On Debian linux, Installing Ollama is a one liner, just enter the following in your terminal

curl -fsSL https://ollama.com/install.sh | sh

Yup, that is it, move on to using Ollama

1.2 Windows and MAC

Just go to https://ollama.com/, download it, and run the installer ! you are done

Optional: Changing the models directory

AI models can be very big, sometimes half a terabyte, for example, at the time of writing, deepseek model was around 700GBs, but the disk you are copying to needs to be an SSD if you want reasonable speed.

So here are the steps

• sudo systemctl stop ollama
• sudo mkdir -p /hds/2tb/ollama_models (Wherever you want the data to be)
• sudo chown ollama:ollama /hds/2tb/ollama_models (Whatever your username is)
• sudo mkdir -p /etc/systemd/system/ollama.service.d/ (This is the override conf)
• sudo nano /etc/systemd/system/ollama.service.d/override.conf
  and add the following to it
  [Service]
  Environment=”OLLAMA_MODELS=/hds/2tb/ollama_models”
• sudo systemctl daemon-reload
• sudo systemctl start ollama
• sudo systemctl show -p Environment ollama

Using it !

Using Ollama is simple, just open your terminal window or command prompt , then activate your conda environment (Or venv) , and run the following command, for the sake of this example, I will run

ollama run llama3.2

llama3.3 with its 70 billion parameters will require a minimum of 64GB of ram, so don’t try that unless you have the RAM for it ! for comparison, 3.2 has 2 billion, which is around 3% of 3.3

It should now probably download about 2GBs of data (The model llama3.2 has around 2 billion parameters) And you are done, Now you can ask it anything

For example, create an article for me explaining this and that,

Once done, just enter “/bye” to exit the ollama prompt and quit the session

If you want to for example clear the context or do anything else, just use the command /? for a list of commands

Now, you have used the lama3.2, but on this ollama models page, you will find that there are many others that you can use !

Others include models that help you with coding, or models that are more targeted towards chat-bot QA, either way, you should take a close look at them, even if for the fun of it

Is ollama running ?

Just visit (http://localhost:11434/) in your browser, and you should see the message (Ollama is running)

What models have been installed (ex: ollama pull llama3.2)

ollama list

Remove installed models from system

Start by listing the models like the above, then you can remove them with rm

ollama rm llama3.2

Modifying a model’s settings

An example of what you may want to modify may be for example If you have a GPU, but you do not want it to be used by Ollama, to do this, you will need to create a model file, the steps to creating this file for llama 3.2, (The small one) are as follows

# Copy the llama 3.2 base file
ollama show llama3.2:latest --modelfile > ~/cpullama3.2.modelfile
# edit the file ~/cpullama3.2 and edit the FROM line to read
FROM llama3.2:latest
# go to the parameters section, and add the parameters you need
# In our case, PARAMETER num_gpu 0
PARAMETER num_gpu 0
# Create your custom model
ollama create cpullama3.2 --file cpullama3.2.modelfile

The last command above resulted in the following output

transferring model data 
using existing layer sha256:dde5aa3fc5ffc17176b5e8bdc82f587b24b2678c6c66101bf7da77af9f7ccdff 
using existing layer sha256:fcc5a6bec9daf9b561a68827b67ab6088e1dba9d1fa2a50d7bbcc8384e0a265d 
using existing layer sha256:a70ff7e570d97baaf4e62ac6e6ad9975e04caa6d900d3742d37698494479e0cd 
using existing layer sha256:966de95ca8a62200913e3f8bfbf84c8494536f1b94b49166851e76644e966396 
using existing layer sha256:fcc5a6bec9daf9b561a68827b67ab6088e1dba9d1fa2a50d7bbcc8384e0a265d 
using existing layer sha256:a70ff7e570d97baaf4e62ac6e6ad9975e04caa6d900d3742d37698494479e0cd 
creating new layer sha256:650ff8e84978b35dd2f3ea3653ed6bf020a95e7deb031ceae487cdd98dedc2e3 
creating new layer sha256:f29c86d4cf6a4072deefa0ff196b7960da63b229686497b02aad4f5202d263ea 
writing manifest 
success 

Above, although you simply created a “model” by copying the existing model’s config file ! nothing more nothing less

Ollama API

So, above, your terminal allowed you to chat with the model, much like what you do when you open Claude or ChatGPT, if you want to access things via API, here is how.

Disable Ollama

To make sure Ollama does not come up at boot, execute the following two commands

sudo systemctl stop ollama
sudo systemctl disable ollama

Removing Ollama

This blog has plenty of posts about AI, some are about AI tools, others are about installing AI locally, so this post is where I am putting all the AI stuff I have ever blogged about in one place !

The section Local AI is about creating your own AI server using freely available sources, the API section lists all the services that provide an API that can be used remotely (Most can not be installed locally anyway), and the Online Services is where you can get things done via AI online (That can be used using your browser, whether they provide an API or not, and whether they can be installed locally or not is a different story)

Continue reading “Everything AI – TOC”