Server Stories, Pt 2 - The Hunt for Yellow CAT

Noticed that an accident had happened: this blogpost was supposed to be published months ago - in fact before the previous update. My bad, somehow I thought I had already done this. On the other hand my notes got a bit messy as I forgot to write everything more clearly early on, and then it become happened a long time ago... I'll have to reconstruct and recall from lost memories some details, since doing all the tests again would not feel like being worth it.

In a summary I went on testing how transfer speeds would variate between transferring to and between SSD/HDD and if there was noticeable difference between software RAID 1 setup and separate drives. All my testing went pretty much bogus from the beginning, as I got shocked by my transfer speed through the network becoming just a fraction of what it used to be earlier though the network...

From Dr. Serverstone's jungle diary:
"Mon  6 Jan 15:35:15 EET 2020:
Transfer to SSD (second test below) is basically just as fast as on HDD (first test below). Networking is the bottleneck now. There is a setup where the sending computer is connected to switch and then router, while the server is hooked directly to the router. This might be an issue.

Transfer package 15.9 GB. Transfer speed was maybe around 11 MB/s.

Test 1, time cp to HDD: 22:45.97 min
Test 2, time cp to SDD: 22:41.18 min

What the fart has happened with the speed though?!?"

The diary doesn't directly tell me whether that was a copy on RAID drive, but as far as I know the RAID was disabled at this point. At this point I had installed a new CISCO Ethernet Switch, and I started to suspect it actually caused some serious slowdown even while that should have not happened.

Is this the culprit for lost networking speed?

More notes from Dr. Serverstone's jungle diary:
"Mon  6 Jan 16:10:37 EET 2020:
New test - both computers straight to router. Transfer speed to SSD up to around 56 MB/s with dropdowns to around 25 MB/s probably during file switch. Package size 8.2GB. Transfer to HDD seemed more or less similar, however, at least dropdowns seemed less deep.

SSD test: 3:59.37
HDD test: 2:57.62

Transfer speeds were now largely as expected regarding to networking. As a surprise though, HDD transfer this time was 25% less than the SSD transfer. This would suggest that the motherboard connection employs SATA 1 connection while the PCIe card with SATA adapters go with SATA 2. This also means that the SSD seems speedwise practically useless on this drive."

Of course there could have been some random variation as well, since there were no repeated testings done. The old computer's resources were considerably limited, so sometimes the computer just started hickuping with background processes that would be nothing for a modern computer. This was of course one important part of my testing too.

Server Jungle diary continues:
"New test - both computers connected straight to router. Transfer speed to SSD up to around 56 MB/s with dropdowns to around 25 MB/s probably during file switch. Package size 8.2GB. Transfer to HDD seemed more or less similar, however, at least dropdowns seemed less deep.

Transfer speeds were now largely as expected regarding to networking. As a surprise though, HDD transfer was 25% less than the SSD transfer. This would suggest that the motherboard connection employs SATA 1 connection while the PCIe card with SATA adapters go with SATA 2. This also means that the SSD seems speedwise practically useless on this drive."

So yea, something often overlooked with recommending SSDs as a boostup for an old computer: if it's a computer old enough to only use SATA 1, it's not really of use anymore... My wife's old laptop Lenovo Thinkpad T61 from 2007 (which she btw used no problem till the Autumn, and still occasionally uses - I bought her an used Thinkstation S30 desktop computer from 2013 as a replacement for sort of a late Christmas/wedding gift, so that we could play World of Tanks and some other games together more properly) was boosted with an SSD drive and there it worked fine and speeded up a lot - but only after BIOS upgrade, since the stock BIOS did not support SATA 2 even while the hardware itself supported it. Although it also must be remembered that an old HDD is often also a lot slower than a new HDD, and these HDDs were brand new Western Digital Red NAS drives.

Server diary continues:
"Mon  6 Jan 16:51:49 EET 2020:
Important general test result:
SSHFS process was constantly below 50% CPU now that there was no software RAID applied to. With the software RAID the CPU was regularly hitting at least over 80% values.

Test with HDD moved straight to HDD. There seemed to be a longish hiatus at the beginning of transfer, after which the transfer rate seemed to jump all the way to around 59 MB/s, before returning to similar transfer rates as before - but the minimum apart from file change drops seemed to be around 55 MB/s.

HDD: 3:27.71

Slower than with HDD in SATA adapter but faster than SDD on mobo. Weird results. Testing yet another part with bigger 15.9 GB package. Previous packages were those 8.2 GB.

HDD on mobo SATA, 15.9 GB package: 4:59.34

This transfer result is still faster than earlier tests for the same package on RAID and direct to SSD. A bit puzzling, might be certain degree of random variation included... For confirmation, maybe one more test with this specific setup is recommended.

Last test with this networking is a transfer to a solo HDD on PCIe SATA adapter. Transfer speed shows huge fluctuation now constantly from 14 to 57 MB/s or so with this 15 somethin GB package. In the end the transfer went on stable high rate again, so all this fluctuation might be due random factors with networking or computer processing.

HDD on SATA adapter, 15 GB package: 7:45.14

Overall I should've made multiple times the same test and take the average.

HDD on SATA adapter, 8 GB package: 2:42.61

Nevertheless, now the SATA adapter package on HDD won again with the smaller package - the fastest time of all in fact. I guess I'll simply plug in both drives to the adapter as they used to be, adds to clarity both inside and on operating system as well.

However, I should yet confirm that the speed is sufficient if both computers are going through the Ethernet switch. And that is where the disappointment comes up. With the CISCO switch the transfer speed just can't get past 12 MB/s or so. Not even while the both computers are in the same network.

HDD through switch, 8 GB package: 12:37.09

Switch instructions say to connect the switch by address 192.168.1.254, but that doesn't work unless if my computer is in the same subnet...which is not true. My computer is in 192.168.10.x, so how do I access that address? The switch should be suitable for up to a gigabit transfer rate (125 MB/s), but now it's only running apparently with 100 megabit speed. This should be configurable, but I can't access the web utility now...

Solution to access the subnet was by changing the LAN settings from router. Then I could access the switch and set it go back to the wanted subnet, after which I set the router settings back again. Unfortunately this flushed my old IP address settings as well down the drain, so some of my manually made scripts will now have incorrect IP addresses. Oh well, it's not so big thing I guess. Maybe some of the biggest things are about setting couple old LAN games working again by the correct IP such as Borderlands.

In addition obviously my connections with my server got quite messed up since I didn't shut down everything before changing settings..."

IP address calculations with all the subways and gateways can actually mess up my head quite nicely when the numbers start to roll around, but it was then found out that there was not so much to adjust with the switch settings manager. Or should I rather say, there was no need to change much.

Diary comes to its climatic plot twist:
"Mon  6 Jan 19:25:26 EET 2020:
Then I found out that Port Settings from the Switch allowed only 10FDX speed to my main computer. The unthinkable had happened: I had used an old Ethernet cable was which was of Category 5, not even 5E. CAT 5 is basically obsolete and it should not be able to reach higher than theoretical maximum of 100 Mbps speed. Now that I changed cables to CAT 5E the Switch shows port speed as 1000FDX, which is more appropriate.

Just. Single. Letter. Difference. One doesn't encounter CAT5 much nowadays.

However, to my shock when I start my copy process once again it shows transferring to only hundreds of kilobytes per second! For a moment I thought that'd been now megabytes due improved setup....fortunately, that apparently was just some temporary delay in processing, as after few moments the actual speed rose up to the preferred 50 +/- something MB/s again. Finally the speed seemed more or less expected for the home setup.

HDD 8GB with Switch and CAT 5E: 3:36.57
HDD 15GB on the same setup: 5:21.64

Then finally I plugged in my server directly to router again with my desktop client behind router & switch. This is because the switch and desktop are in another room with other computers, and the cablings would go awkwardly should I connect also the server to the switch. Transfer speeds seem all sufficiently decent ~50 MiB/s +/- something again also this way. Might in theory be marginally faster to have both computers running through the switch, but to be honest it doesn't matter that much anymore, and it might just as well be all random variation in every transfer anyway. In addition, at least the final test yielded in any case the fastest transfer results.

2:53.58

For the end I'll just yet reconnect both HDDs on my obsolete Fujitsu-Siemens server back to the SATA adapter card and close the case for now. Next stage would be setting up the server accessable for all local computers and users with corresponding folders and so on. This I'm not going to do today, since I already wasted one day of a well deserved extended weekend day."

Again the diary doesn't define which package was that fastest transfer; it would make sense for it to be the 8GB package, but if it really was the fastest transfer, there was above one faster transfer case with 8GB. I probably was just being sloppy with my markings, since like said my notes were largely a mess. In any case, I was really happy after substantial amount of frustration to realize it was the old cable that caused slow transfer, and I didn't feel like testing RAID etc. again anymore.

Speaking of which though, the tests yielded one more important general test result:
"SSDFS [service process for file transfer through SSH] was constantly below 50% CPU now that there was no software RAID applied to. With the software RAID the CPU was regularly hitting at least over 80% values."

This means that the software RAID application mdadm actually does take a fair deal more of processing power - too much from this cake. In all documents I read about mdadm it was stated that in theory a software RAID might take a bit more CPU resources than a hardware RAID, however that would not have any practical significance... Well surely so if the computer used is less than 5 years old and not 15 years old! Nevertheless, I find it actually awesome that such aged device is still actually usable for real, even while in practice one could likely get something more powerful with less watts consumption by dumpster diving today.

Server Stories, part 1

An associate of the Retro Corner

I have a corner desk in our living room with few old computers, and one of them is a Fujitsu-Siemens Esprimo P. This Pentium 4 CPU equipped cream coloured mini-tower computer is an ordinary computer from late 2000's. An exact model and year of release are not in my knowledge at the moment; it looks outside like Esprimo P5905, which also used the same D2151-A1 motherboard. The 5905 was at least originally shipped with Windows XP though, whereas this unit came with Vista, which was not even released yet in 2005 when this review (https://www.alphr.com/desktop-pcs/28016/fujitsu-siemens-esprimo-p5905-review) was written. Probably at least essentially the same computer nevertheless, just presumably sold couple years after first release as a low-price model. As a side note I actually bought a Fujitsu-Siemens computer in 2005 as my first own genuinely modern computer, and I was quite happy with it, but that was different model and overall it is another story.

The computer barely fits into retro category, but I'm also not using it for retrocomputing purposes; this is my new home server. Or at least it is intended to be one. So far I have installed some new hardware on it (960 GB SSD drive and two 4 TB hard drives for storage purposes through a PCIe SATA-II card), but most importantly I have installed new operating system on it: openSUSE Leap 15.1 Linux.

A fake server hiding under family art.

OpenSUSE Leap 15.1 is stated to require as minimum system requirements 64-bit CPU, 1 GB of RAM (2 GB recommended) and 5+ GB disk space (https://opensuse-guide.org/installation.php). My old F-S computer's specs are easily above minimum: the processor is 64-bit version and there is whopping 3 GB RAM on it. It should be possible no problem to get even 64-bit Windows 10 installed on this, but I just wouldn't want to actually using that combination. However, installation of a modern OS did not go through by default settings. I made my boot DVD and the computer booted with it no problem (the installation even had a fancy winter/christmas theme if I recall correctly for the first try, as the computer's Real-Time Clock battery was missing and hence the time was well off), but when I tried to start installing, the computer worked on quite a while until eventually process counter froze at 100% and nothing ever happened.

Shots into troubles

In a typical modern installation guide like in the homepage linked above there is no troubleshooting, and the mostly self-explanatory installation process just goes through like a dance when you're the prom queen. Of course most people wouldn't go install new operating systems to old computers anyway, so the inevitable issues would be met by only us few...in any case I had to figure out what went wrong.

The thing was with BOOT menu settings for Kernel Default, which had to be changed first by pressing F5 on bootup/install menu with the DVD's GRUB startup. This computer was old enough to either not support APIC (Advanced Programmable Interrupt Controllers) at all or at least not the version expected, so I had to turn it off for installation. Later on I had to edit my /etc/default/grub file probably due this, otherwise the computer would not shut down by software commands, but after that the ATX software power control seemed to cause no issues.

Anyway, after this setting change I was able to install the new Linux distro. I've never had openSUSE, so it would be a new learning experience after having mostly used Debian based distros. Especially as I opted to not use GUI at all - there wouldn't be excess amount of computer resources to waste, and for most of the time I was not intended to be exactly sitting next to a monitor with this computer anyway. SSH would be the way of control for this partner in computing, straight from my main workstation the next door. I'll just need to install SSH server (sshd, actually SSH daemon) and set my firewall settings properly, including swapping the default port 22 to something say 5 digit numer for additional security. First things with new Linux distro comes up: to install stuff I can't use apt-get commands but zypper, and the default firewall application is not UFW but runs just by firewall-cmd.

This being said, about not having GUI, I do recommend installing fish shell for making life with CLI a bit easier - even if using terminal just occasionally. Fish shell gives a bit Powershell-like option to browse available commands by TAB and does other usability assistance features such as adding colours to terminal text a bit similarly as programming text editors do (ie. IDE applications). Default Bash might be powerful for sure, if you know how to use it, but that is essentially just an enchanced version of 1970's technology. Barebone Unix shell can be really daunting to use if you have no long experience, no tutor to support you, or no proper thick manual within reach.

Yea, you can google around every command separately if you have another computer around (or use remote access), but googling is just a terrible mess in comparison to having any of those three. Googling gets stuff done, but it's really like going to nearby woods to fetch some berries or mushrooms; you might know how to find them, there might be fresh stuff around to pick, and then it's just a matter of time, but it sometimes takes a lot of time to find what is needed and usually there is need to search for multiple places at once. Not to mention that lots of data in Internet is already well rotten with their due date already ten years behind or it's worm eaten to begin with, and it can be hard to see from afar which one is a good pick - at least with books it's possible to judge by the cover (or front pages) when it was originally released and that books in theory are more prone to have more certified facts. That of course is not always true either, but now I'm getting sidetracked.

Server is up and sitting

So now I can access my fancy home server from another computer in my local home network (actually I set it up to be accessable also from outer web, but that needs to get more strict security rules on how who and where). What then? What to do with a home server? Many people don't really seem to have a clue what to do with a server, and I suppose it is understandable, especially as I can't do just whatever with this. Nevertheless, let me think of few options.

1. File server. I can set a private file sharing server that can also provide network drives for all local devices we have at home. Alternatively it could be used for transferring data to other people. Sure there are all kind of free cloud services available, but if you care about privacy the free ones become a bit less tempting. On the other hand if you need space for hunderds of gigabytes of data or even terabytes (video material in my case hogs probably like 80% of my used terabytes), the free services are just not sufficient and paid services might actually become pricey with higher storage uses.
2. Web page server. Well I'd lack DNS support, so people would need some fancy http://111.111.111.111/sakariaania style address to access it, but I might be able to use it myself for something like own data, easy checkup whether the server is still up, and could provide direct links for people who could need it. Would always amuse me to have own www server at home.
3. Surveillance camera. "Why would you need that?!?" Yea, I don't have nothing to steal and I'm always at home anyway, right? Any home could be broken into even if there is nothing really valuable inside, and getting photos of the intruders and time of the event might help a lot on figuring out the possible crime. Like risk of fire, such things are unlikely, but it's still best to have some preparation for this unfortunate thing to happen. On more practical side though, I could check if there is any mail arrived in case I'm actually not at home.
4. Mail server. Okay, for this too to work properly I'd need some extra services set up, and I might be quite content with having some other email addresses...but at least it would make it possible for my server to send email notifications if some service fails - or even to build up some kind of MFA system.
5. Remote control. Technically if there is anything I can control by a computer and I wish to have remote access for controlling that, a server with just SSH access could become a tool for that.

There'd be of course various other possibilities, but those would hit as something realistic and usable.

In addition to this all, for me it is also largely just about testing and learning. If I'd really need a server, I could more easily set up one, and also I may encounter things that can help me understand things I encounter at work. Hands-on method teaches a lot more to me than being given server IP and credentials for remote logging some server so I can do some Active Directory management for instance. That doesn't really tell me 'what' is the server for real. By setting one up I'll get a bit better impression - even if no enterprise environment would rely on setting up a server on 10+ years old home computer, especially as nowadays hardware servers are less and less frequently set up when you can just have virtual servers.

RAID over Sakariaania

For now I'm only going to go for the file server option, as that I actually need. Since I bought two identical 4 TB hard drives for this specific purpose, I also felt like testing RAID 1 state with them. For a moment I thought my PCIe SATA card with RAID controller would actually work as a hardware RAID, but then I found out that these are deemed as "fake raid" devices. Not only that, but especially on Linux it is recommended to use software RAID setup instead both because the "fake raid" could cause technical issues (including reduced reliability: should another drive or the computer itself fail, the other drive might not be recoverable on another platform) and because Linux's own software RAID application is supposedly very good and robust. It just might, maybe, take slightly more resources than the hardware controller supported "fake raid". Is my old Esprimo up for the task?

SATA-II PCI-Express Card still unpacked.

First shock came after installing the card and drivers. The card itself was supposed to have its own BIOS for automatically setting up a RAID that could be started by pressing a function key during computer self-test. I guess the key was correct, since pressing it froze the whole system. When I merely started the computer I could not find the drives at all. Oh dear, did not the card work on this computer at all? Silly me, however, I forgot that the new HDDs were completely unprepared. I went on my other computer, created partition tables and formatted the drives. After that I could find the drives on my Fujitsu-Siemens as devices but not from the file system. For a little moment I also had forgotten that on Linux like this the drives would not be automatically mounted either...such corruptive effects all those modern convenience features can cause. So after partitioning, formatting and mounting the drives could be found on the server computer - as their separate entities.

The Linux application 'mdadm' is an application with which you can set up a software raid on whichever partitions on whichever drives available, so that is the recommended option for setting up a software RAID. I can already tell that it worked no problem and wasn't too hard: just needed couple commands to assemble the RAID and then to enable it. As another option I'd had potentially 'dmraid' application with which it should have been possible to work with the PCIe card itself, but I ended up not even trying that for real. Like referred to in the previous paragraph, I was never able to reach this card BIOS on this computer, which was a bit of a bummer. Fortunately, the disappointment faded away gradually after I first found out that this 'dmraid' could have set up the necessary things, and even more after I found out this that it's not likely a good setup anyway.

If I'll now check on my server terminal for instance disk free status on all drives by command 'df', I'll have this RAID disk called /dev/md0. After a moment I can mount it as a network drive on my main computer. Mission successful? Not quite. This is just a start. On the other hand, because this is also a test, I will yet change my setup with drives a bit.

First test of copying was not perfectly impressive though. The computer is rather silent normally, but when I for the first time tested copying few GB worth of data over the SSH file transfer protocl just for testing functionality, CPU hit 100% and computer fan jumped from like 1100 to 2600+ RPM and that made it rather loud. It was night time too...oops. Copying locally from drive to drive the CPU hogs "only" around 80% and doesn't make much sound. Normal idling temperature of the fan is like 40-50 C, yet this 100% CPU transfer situation took the temperature close to 80 C - but barely further. So temperature was not that bad, the computer made it ok and main issue was just the fan. I might consider either swapping the fan or adding another silent one to make cooling go better at some point.

Nevertheless, this test made me reserved with the RAID setting. Was the software RAID too heavy after all or was it just the SSHFS? On my subsequent file transfers there was no similar heavy fan load for whatever reason, and when I checked the process resource drainages, actually it was not mdadm which took resources but sshfs. Also it took lots of resources to transfer to the non-RAID drive as well, so I suppose the software RAID is not really a problem. It just is heavy process for that computer to transfer the files over the connection. Although that is somewhat difficult to say, since the md0 synch of RAID drive seems to cause multiple processes that can take few % CPU even when nothing is really done, so that might stack up on unnecessary drains on an old computer like this. I'll probably eventually remove the RAID after testing - it's not really needed after all, but I had never tried it. Instead I can just use scheduled backups every now and then.

Linux command line has also this convenient command 'time' to check up duration of any command. Therefore I could test transfer speed. So by hitting the command "time cp [/source/path] [/target/path/through/sshfs]" I got following results on two different tests:

From local HDD to server RAID HDD.
real    5m55,582s
user    0m0,212s
sys    0m25,496s

From local HDD to server SSD.
real    5m31,433s
user    0m0,238s
sys    0m25,763s

My test package was 15.9 GB worth of video files. Transferring to SSD was a bit faster than to the HDD RAID as was expected, although I'm also sure the SSD can't use its full potential on that computer. Also I'm actually thinking the motherboard SATA is only SATA-I whereas the PCIe card should at least in theory provide SATA-II connectors. Although I guess 45-50 MB/s might be fairly decent transfer speed for this kind of a setup. I'll have yet some more tests to go with for further conclusions. Also I might yet test if I can actually get the RAID set up by 'dmraid' command set. We shall see when the next part comes.

Can't deny though - this kind of a warrior pair is more or less asking to join a RAID.