Michael<p>Putting the data roots for each buildah container into a separate directory on the shared CephFS volume did work and reduced the overall build time for the FluentD container from 23 minutes to 15 minutes. So the shared dir was part of the problem. But it's still rather slow, and I'm still pretty sure it's because of the VFS driver and CephFS. So let's see whether I can come up with something better.</p><p><a href="https://social.mei-home.net/tags/HomeLab" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>HomeLab</span></a> <a href="https://social.mei-home.net/tags/Ceph" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Ceph</span></a> <a href="https://social.mei-home.net/tags/buildah" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>buildah</span></a></p>
#ceph
6 posts6 participants0 posts today
Are you running #OpenStack and #Ceph? Share your story at #Cephalocon In Vancouver on October 28! The CFP is currently open and closes this Sunday, July 13 at 11:59pm PT.
Seems some of my disks have seen a few writes! `smartctl -a ${DEVICE}` stats for the #Ceph and virtual machine host cluster here.
The 2TB SSDs are OSDs in Ceph, the others are OS disks or local VM cache disks. Ceph is moaning about the disks in "oxygen" and "fluorine".
Numbers here are Total Drive Writes in GB.
```
$ for h in *; do for d in ${h}/tmp/*.txt; do lbas=$( grep -F Total_LBAs_Written ${d} | cut -c 88- ); if [ -n "${lbas}" ]; then dev=$( basename "${d}" .txt ); dev=/dev/${dev##*-}; sect=$( grep '^Sector Size' ${d} | cut -c 18-21 ); printf "%-20s %8s %6d %s\n" ${h} ${dev} $(( ( ${lbas} * ${sect} ) / 1024 / 1024 / 1024 )) "$( grep '^Device Model' ${d} )"; fi; done; done
beryllium.chost.lan /dev/sda 0 Device Model: WD Green 2.5 240GB
boron.chost.lan /dev/sdb 76370 Device Model: Samsung SSD 870 EVO 2TB
carbon.chost.lan /dev/sda 113892 Device Model: Samsung SSD 870 EVO 2TB
carbon.chost.lan /dev/sdb 157993 Device Model: Samsung SSD 860 EVO 2TB
fluorine.chost.lan /dev/sda 111939 Device Model: Samsung SSD 870 QVO 2TB
helium.chost.lan /dev/sda 100476 Device Model: Samsung SSD 870 QVO 2TB
hydrogen.chost.lan /dev/sda 184564 Device Model: Samsung SSD 860 EVO 2TB
hydrogen.chost.lan /dev/sdb 58602 Device Model: Samsung SSD 870 EVO 2TB
lithium.chost.lan /dev/sda 0 Device Model: WD Green 2.5 240GB
magnesium.chost.lan /dev/sda 0 Device Model: WD Green 2.5 240GB
neon.chost.lan /dev/sdb 146926 Device Model: Samsung SSD 860 EVO 2TB
nitrogen.chost.lan /dev/sda 99473 Device Model: Samsung SSD 870 EVO 2TB
oxygen.chost.lan /dev/sdb 108748 Device Model: Samsung SSD 870 QVO 2TB
sodium.chost.lan /dev/sda 0 Device Model: WD Green 2.5 240GB
```
Replied in thread
I'm a glutton for punishment this week… I'm poking the octopus again, moving to Ceph 18.
So far, we're doing the monitor daemons. A fun fact, they switched from LevelDB to RocksDB some time back and in Ceph 17, they dropped support for LevelDB.
I found this out when updating the first monitor node, it crashed. Refused to start. All I had was a cryptic:
ceph_abort_msg("MonitorDBStore: error initializing keyvaluedb back storage")
Not very helpful. The docs for Ceph 17 (which I didn't read as I was going from 16→18 direct, which the Ceph 18 docs says you _can_ do), merely state:
> LevelDB support has been removed. WITH_LEVELDB is no longer a supported build option. Users should migrate their monitors and OSDs to RocksDB before upgrading to Quincy.
(https://docs.ceph.com/en/latest/releases/quincy/)
How? 0 suggestions as to the procedure.
What worked here, was to manually remove the monitor node, rename the /var/lib/ceph/mon/* directory, then re-add the monitor using the manual instructions.
https://docs.ceph.com/en/latest/rados/operations/add-or-rm-mons/
docs.ceph.comQuincy — Ceph Documentation
Huh, can I manually point two persistent volumes at the same CephFS filesystem?
I just realized that a huge part of my plan depends on this
#Kubernetes #Ceph