This procedure describes the standard installation of the base OS of an OCP server as done by RapidSpace. Partners should follow this procedure but can use another one if they want.
1. Goal
The goal is to install Debian 10 on all OCP servers of a rack. The OS must be installed on the small SSD disk so that the big SSD disk can be used totally by the Rapid.Space VM. At the end of the procedure, all servers are installed with SSH access.
2. Prerequisites
1. Hardware
Here is the list of hardware you need to complete the procedure:
- 1 raspberry pi
- 1 RJ 45 cable
- all the OCP servers you want to install
- 1 switch for the OCP server
- all the cables to connect the switch and the OCP servers
2. Network
In order to complete the procedure, you will need internet access. The raspberry pi must be able to connect to the internet through wifi.
The OCP servers must also have internet access through IPv4 during the procedure. This procedure is written to give the server internet access through the raspberry pi. If you want to use the regular internet access (internet access through the switch and the datacenter internet provider), it's possible but the procedure is more complex and not written in detail here.
3. Prepare raspberry device
We use a raspberry device that will be used to do DHCP server and TFTP server during the installation. The raspberry will contain all the scripts needed to install the OS.
Note: The raspberry can be replaced by the switch in the rack if it runs Linux (this is the case for our standard AS5812 switch) or even by any machine inside the rack. The advantage in using the raspberry is that it works no matter the current environment.
Install the raspberry following rapidspace-Create.Raspberry.For.Remote.Maintenance then, as root:
cd /root
apt install ipmitool dnsmasq-base git
git clone https://lab.nexedi.com/nexedi/rapidspace-autoinstall-os
4. Correctly plug the network cables
In order to do the installation, the raspberry pi needs to be plugged to the BMC network interface to do IPMI commands. The BMC network interface is usually the RJ45 interface directly on the OCP server motherboard. If your server doesn't have a RJ45 interface soldered in, the BMC interface is then the Mezzanine network card. In this case, you will need to plug the RJ45 cable to a RJ45/SFP adapter and plug the adapter into the SFP+ port of the Mezzanine card.
For the raspberry to provide internet access, you need to run the following commands as root:
echo 1 > /proc/sys/net/ipv4/conf/all/forwarding
update-alternatives --config iptables # chose legacy iptables
iptables -t nat -A POSTROUTING -o wlan0 -j MASQUERADE
In this configuration, you will use the same network cable to do IPMI commands and to provide internet access to the machine. You will need to plug each machine after the other in order to install all of them.
Note to use datacenter internet access (through the AS5812 switch) during the installation:
Sometimes, it is desirable to use the regular internet access to install the machine. In this case, you will plug the SFP+ cable between the switch and the machine. In this case, you will run the dnsmasq script in the switch instead of in the raspberry pi. THis is for advanced users only.
5. Adapt scripts to your location
cd /root/rapidspace-autoinstall-os
cp dnsmasq_hydro66.sh dnsmasq_XXX.sh
cp preseed_hydro66.cfg preseed_XXX.cfg
You need to copy the scripts as you can see above. You can give them a name representing the location of your rack. As you can see the script dnsmasq_XXX.sh will do the following things:
- download the needed iso files if needed
- prepare a special grub config file that will be able to recognize the server depending on their MAC address
- prepare the network interfaces configuration file for each machine
- launch dnsmasq executable to act as a DHCP server + TFTP server
You need to carefully replace all the variables with what you want in your site. You need also to replace in the linux command line for Debian the name of the preseed script.
Be careful, for DHCP_RANGE if you chose static, then you need to specify each MAC address that can have an IP with --dhcp-host option of dnsmasq command.
You need also to adapt the preseed configuration file for automatic Debian installation. You need to look carefully at the following lines:
- d-i mirror/http/hostname this is the hostname of the APT mirror you want to use
- d-i partman/early_command this is the command executed to select the good disk to install your OS on. Be careful with this command in order to install on the correct disk
- d-i preseed/late_command this command is run at the end of the installation. It will get the network interface file through tftp. You need to use the proper IP.
A good thing to do is looking at the diff between the different dnsmasq and preseed files in the repository.
How to find good value for early command
You will see in the early command something like this:
d-i partman/early_command string set -e; x=`grep -l 488397168 /sys/block/*/size`; x=${x%/*}; x=/dev/${x##*/}; debconf-set partman-auto/disk $x; debconf-set grub-installer/bootdev $x
The number 488397168 is the number of blocks of the disk you want to install the OS into. In order to find the correct value for your server, you should run a first time the debian installer with any value there. When the command will fail, the installer will stop and you will land in the debian installer shell. In this shell, you can find the value by running grep '' /sys/block/*/size. This will show you all the disks with the size. Then pick the smallest number there.
When everything is ready, you should simply launch the dnsmasq script in a terminal (the script should stay opened in the terminal so you can see dnsmasq logs):
cd /root/rapidspace-autoinstall-os
./dnsmasq_XXX.sh
6. Use IPMI to go to the BIOS
Once the IPMI is plugged to the raspberry (either directly or through a switch), you will see that BMC does a DHCP request in the dnsmasq logs. Use this IP address to connect to the IPMI. You can use this command to make sure IPMI access is OK:
ipmitool -I lanplus -U XXX -P XXX -H 192.168.46.XXX chassis power status # replace the XXX with correct values for username/password/IP address
Then, you need to type the following commands to force to boot into bios, then reboot, then connect to serial port:
ipmitool -I lanplus -U XXX -P XXX -H 192.168.46.XXX chassis bootdev bios
ipmitool -I lanplus -U XXX -P XXX -H 192.168.46.XXX chassis power cycle
ipmitool -I lanplus -U XXX -P XXX -H 192.168.46.XXX sol activate
Note about IMPI passwords
By default, the username/password for accessing IPMI are admin/admin. The capri servers need the password to be set before using IPMI. As soon as server IP address is known the command to set the password is:
ipmitool -I lanplus -U admin -P admin -H 192.168.46.XXX user set password 2 nexedi1234
Then you can call above IPMI commands by using admin/nexedi1234 as username/password .
7. Configure BIOS
You must be in UEFI mode and you must select PXE through IPv4 as your first boot device (be careful if you have several interface to use the good one, there is the MAC address of the interface to help you).
Note: UEFI mode can be in "CSM configuration" menu of BIOS.
8. Do installation
Use IPMI to reboot the system again and follow the installation progression through the serial port:
ipmitool -I lanplus -U XXX -P XXX -H 192.168.46.XXX chassis power cycle
ipmitool -I lanplus -U XXX -P XXX -H 192.168.46.XXX sol activate
When grub shows up, you must manually type "Enter" to select debian installation. Then the rest of the process should be fully automatic. If something goes wrong, you can debug what is happening with the shell inside the installer and modify the preseed files to what you want. Then restart the installation.
9. Test
At the end of this procedure, you should now be able to login with SSH on all the servers of the rack. Make sure it is really the case. Each server must have internet access through IPv4. Make sure, you can ping 1.1.1.1 from all the servers.
10. Next steps
When all the OCP machines are installed with Debian 10, you can follow the procedure described in rapidspace-HowTo.Partner.Setup.Server.Installation.