FreeBSD 10.2-RELEASE 安裝在 GMIRROR 上

因為之前 worren.net 只用了 1 顆 160GB 的 hdd in a thin PC . 所以只能將 /etc 等相關的設定檔進行定期異機backup  .. 這次hdd 整個掛掉 , 花了一周的時間 survey 新機器 / 採購 / 安裝 ... 1 周當然都只是用空餘的時間 ... 但這次被惠雅一句話提醒了我... 幹嘛不換一台好一點兒的機器,  一直在看我修電腦 ... ... 買了 2bay 的 ASUS Vivo PC , 就是要拿來做 RAID 1 .

VivoPC 是採用 intel 平台 . 有支持 RST . 但... 還真的不知怎麼用 ! 我也懶得找 RST in FreeBSD 的文件了 . 決定直接用 FreeBSD 內建的 software . .. 於是乎, 找到以下這篇文章 .. 有完整的介紹 .. step by step . .. 把她節錄一下 , 以免以後她不見了 ..

https://www.ateamsystems.com/tech-blog/installing-freebsd-9-gmirror-gpt-partitions-raid-1/

Home > Tech Blog > HOWTO: Installing FreeBSD with gmirror (software RAID 1) and the GPT partitioning scheme

HOWTO: Installing FreeBSD with gmirror (software RAID 1) and the GPT partitioning scheme

Posted on Saturday, May 5, 2012 under , , , ,
Last Updated April 30th, 2015. This article is really popular, I’ll do my best to keep it up to date! I’ve added notes about TRIM, alignment and GMIRROR (-h) labeling since it was originally posted.

Note: These instructions have been verified to work for FreeBSD 9.1, FreeBSD 9.2, FreeBSD 10.0, and FreeBSD 10.1 .

By default gmirror and the GPT partitioning scheme do not get along.  This is because both GEOM (the provider for gmirror) and GPT write meta data at the end of the disk.

Note: Instructions on rebuiling a gmirror with this kind of setup are at the bottom under Testing & Replacing A Failed Disk.

The concept behind this procedure and "work around" is that instead of mirroring the entire disk, you use gmirror to mirror each of your partitions.

The order of these steps is important to avoid corruption.  I would only do this on a clean install (and these directions are for that).  If you setup a gmirror after the fact on a running machine the gmirror meta data has to get written somewhere, and if you've got a file system on it with data, you risk corrupting it and losing data if it happens to be at the end.

To avoid this we'll setup GPT, create the partitions, create the gmirror providers and finally newfs the appropriate partition.  This ensures nothing will get corrupted.

Step 1: Boot The FreeBSD Installer

Start the FreeBSD install process as normal.  When you are asked about how you want to partition the disk pick "Shell":

FreeBSD 9 installer

Step 2: Setup The Partitions

Once inside the shell here are the commands to setup GPT and slice up both disks.  These examples assume you are doing this on disk 'ada0' and 'ada1' (SATA), if you are using SCSI you'd do da0, da1. We also align the partitions to 1 MiB boundaries which will work with 4k (or advanced format) disks, for more information on this read my post about partition alignment under FreeBSD for 4k disks.

# ---- Setup 1st disk
#
gpart create -s gpt ada0
gpart add -s 128k -t freebsd-boot -l boot0 ada0
gpart add -a 1m -s 8G -t freebsd-swap -l swap0 ada0
gpart add -a 1m -t freebsd-ufs -l root0 ada0

# -- Install boot code to first disk
gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 ada0

# ---- Setup 2nd disk
#
gpart create -s gpt ada1
gpart add -s 128k -t freebsd-boot -l boot1 ada1
gpart add -a 1m -s 8G -t freebsd-swap -l swap1 ada1
gpart add -a 1m -t freebsd-ufs -l root1 ada1

# -- Install boot code to 2nd disk
gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 ada1

Step 3: Setup The Gmirror Providers

Before we proceed we need to have GEOM "re-taste" the partitions so our nice labels show up in /dev/gpt/:

true > /dev/ada0
true > /dev/ada1
# You can check to make sure this worked by running:
ls -l /dev/gpt/

# Output should look similar to this:
crw-r-----  1 root  operator    0, 100 /dev/gpt/boot0
crw-r-----  1 root  operator    0, 108 /dev/gpt/boot1
crw-r-----  1 root  operator    0, 102 /dev/gpt/root0
crw-r-----  1 root  operator    0, 110 /dev/gpt/root1
crw-r-----  1 root  operator    0, 104 /dev/gpt/swap0
crw-r-----  1 root  operator    0, 112 /dev/gpt/swap1

Now we can continue to build the mirror providers for each of our partitions:

# -- Build gmirrors
gmirror label -h boot /dev/gpt/boot0 /dev/gpt/boot1
gmirror label -h swap /dev/gpt/swap0 /dev/gpt/swap1
gmirror label -h root /dev/gpt/root0 /dev/gpt/root1
# -- Load the geo_mirror KLD
kldload geom_mirror

# -- Check status
gmirror status

# -- You should see something like this:
       Name    Status  Components
mirror/root  COMPLETE  gpt/root1 (ACTIVE)
                       gpt/root0 (ACTIVE)
mirror/swap  COMPLETE  gpt/swap1 (ACTIVE)
                       gpt/swap0 (ACTIVE)
mirror/boot  COMPLETE  gpt/boot1 (ACTIVE)
                       gpt/boot0 (ACTIVE)

Step 4: Create And Mount The Root Filesystem

Here we actually format the root filesystem, enabling soft updates (-U).  I also use "-L root" to set the filesystem label but this isn't needed.  Then we mount the new filesystem in /mnt (which is where the installer expects the target filesystem to be).

If you are using SSDs add the -t option to the newfs call below so that TRIM support is enabled from the start.

# -- For HDDs:
newfs -U -L root /dev/mirror/root
# -- For SSDs:
newfs -t -U -L root /dev/mirror/root

mount /dev/mirror/root /mnt

We now need to create the fstab file which will be put into place by the installer:

vi /tmp/bsdinstall_etc/fstab
# Device          Mountpoint      FStype  Options Dump    Pass#
/dev/mirror/swap  none            swap    sw      0       0
/dev/mirror/root  /               ufs     rw      1       1

Exit out of the partitioning shell to resume the setup:

exit

Step 5: Resume The Setup

Continue through the rest of the install as you normally would do.  At the very end when you exit the installer you will see the prompt below:

FreeBSD 9 installer

Pick Yes.

Step 6: Final Configuration

Inside this shell we just need to set a few last minute things to make sure our new OS loads GEOM on boot and also reports the mirror status in the daily report emails.

# -- Make sure gmirror module comes up on boot
echo 'geom_mirror_load="YES"' >> /boot/loader.conf

# -- Enable daily status reporting
echo 'daily_status_gmirror_enable="YES"' >> /etc/periodic.conf

And that's it! Exit out of this shell:

exit

And reboot!

Addendum: Testing & Replacing A Failed Disk

To test your gmirror you can pull a disk out of a gmirror and the server should log a message in dmesg(8) but otherwise function normally.  When you re-insert the disk it will rebuild automatically as gmirror will recognize it (be sure to let it rebuild fully before yanking another disk).

If you have a failed disk that needs replacing simply remove it and insert the new disk of the same size or greater (gmirror will only use the original size though).  If your hardware supports hot-swap this can be done while the server is running.  With the new disk in place we'll need to re-partition in like we originally did above, tell gmirror to forget the old disk's partitions and add the new disk's partitions to the mirror containers.

Let's assume for this example that ada1 has failed.  Once the new disk has been inserted you'll first want to make sure it's blank by running:

gpart show ada1

If anything but a blank listing comes up (and your replacement disk is new) think long and hard! Make sure you've got the correct disk and device name!

If you are using a recycled disk that isn't blank, first blank out it's partition (again, double check the device name!):

gpart destroy -F ada1

Now we'll partition the disk the way the originals are (above), adjust for any differences in your own partitioning:

gpart create -s gpt ada1
gpart add -a 1m -s 128k -t freebsd-boot -l boot1 ada1
gpart add -s 8G -t freebsd-swap -l swap1 ada1
gpart add -t freebsd-ufs -l root1 ada1
gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 ada1

Have FreeBSD re-taste the /dev/gpt/ device names:

true > /dev/ada1
ls -l /dev/gpt/

Now for each parition tell gmirror to forget any disk's partitions which aren't currently present and add the new partitions to become mirrored (remember we're gmirroring each parittion individually instead of each disk so GPT can work happily with gmirror):

gmirror forget boot
gmirror insert -h boot /dev/gpt/boot1

gmirror forget swap
gmirror insert -h swap /dev/gpt/swap1

gmirror forget root
gmirror insert -h root /dev/gpt/root1

Finally check the status of the rebuild:

gmirror status

More information can be found at the FreeBSD Handbook's page on RAID1 mirroring.

18 thoughts on “HOWTO: Installing FreeBSD with gmirror (software RAID 1) and the GPT partitioning scheme”

  1. rizza
    i try to mirroring with freebsd 10 but i found a problem while insert mirror root.
    gmirror insert root /dev/gpt/root1
    problem = gmirror: provider gpt/root1 too small.
    in my case root ada0 and ada1 had same size.
    please help me ass soon as possible

    thank you

    1. Adam Strohl Post author
      Hey Rizza, it sounds like when you created the partition it got made too small as the error implies, can you show me the output of ‘gpart show’ ?
  2. Lost
    This does not work on FreeBSD 10.

    Please provide actual instructions to “Resume the setup”. You completely miss the actual partitioning in the installer, it’s not even mentioned. So how do I handle disks inside the installer?

    Thanks…

    1. Adam Strohl Post author
      I haven’t had a chance to update the instructions yet, the title of the article doesn’t claim it to work for 10.

      Regarding missing “partitioning in the installer” the whole point is you cannot use the installer to do it and need to manually partition, thus all the instructions above thew “resume the setup” portion where you manually partition the drive.

      The installer’s partitioning system cannot do these tasks which is why I wrote the instructions in the first place.

    2. Krissada Jindanupajit
      It work perfectly on FreeBSD 10 (GENERIC)…

      After you mount all new file systems to /mnt, just type “exit” and it will resume the setup.

      
      root@pbx:~ # gmirror status
      Name Status Components
      mirror/boot COMPLETE ada0p1 (ACTIVE)
      ada1p1 (ACTIVE)
      mirror/swap COMPLETE ada0p2 (ACTIVE)
      ada1p2 (ACTIVE)
      mirror/root COMPLETE ada0p3 (ACTIVE)
      ada1p3 (ACTIVE)
      mirror/var COMPLETE ada0p4 (ACTIVE)
      ada1p4 (ACTIVE)
      mirror/tmp COMPLETE ada0p5 (ACTIVE)
      ada1p5 (ACTIVE)
      mirror/usr COMPLETE ada0p6 (ACTIVE)
      ada1p6 (ACTIVE)
      root@pbx:~ # uname -a
      FreeBSD pbx 10.0-RELEASE FreeBSD 10.0-RELEASE #0 r260789: Fri Jan 17 01:46:25 UTC 2014 root@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC i386
      root@pbx:~ #
      </code>
      
      </section>
      <div class="reply"><a class="comment-reply-link" href="https://www.ateamsystems.com/tech-blog/installing-freebsd-9-gmirror-gpt-partitions-raid-1/?replytocom=113#respond" rel="nofollow">Reply</a>↓</div>
      </article></li>
      </ol>
      </li>
      	<li id="li-comment-114" class="comment even thread-even depth-1"><article id="comment-114" class="comment"><header class="comment-meta comment-author vcard"><img class="avatar avatar-44 photo" src="https://secure.gravatar.com/avatar/79ebd4c340e717d6bef8d592ab7c996c?s=44&amp;d=mm&amp;r=g" srcset="https://secure.gravatar.com/avatar/79ebd4c340e717d6bef8d592ab7c996c?s=88&amp;d=mm&amp;r=g 2x" alt="" width="44" height="44" /><cite class="fn">Krissada Jindanupajit </cite><a href="https://www.ateamsystems.com/tech-blog/installing-freebsd-9-gmirror-gpt-partitions-raid-1/#comment-114"><time datetime="2014-04-25T18:43:50+00:00">April 25, 2014 at 6:43 pm</time></a></header><section class="comment-content comment">For the partitioning, if you want /usr /var and /tmp in different partition from / just
      setup your additional partitions ….
      <code>
      # Follow Adam's instruction but when you create partition, just do what you want
      # (Replace <b>X</b> with <b>0</b> for <b>ada0</b> and then <b>1</b> for <b>ada1</b>)
      ...
      ...
      gpart create -s gpt ada<b>X</b>
      gpart add -s 64k -t freebsd-boot -l boot<b>X</b> ada<b>X</b>
      gpart add -s 8G -t freebsd-swap -l swap<b>X</b> ada<b>X</b>
      gpart add -s 2G -t freebsd-ufs -l root<b>X</b> ada<b>X</b>
      gpart add -s 2G -t freebsd-ufs -l var<b>X</b> ada<b>X</b>
      gpart add -s 1G -t freebsd-ufs -l tmp<b>X</b> ada<b>X</b></code>
      
      # without -s, gpart will use the remaining of your disk
      gpart add -t freebsd-ufs -l usr<b>X</b> ada<b>X</b>
      
      <code>...
      ...
      </code>
      
      And when you build the mirrors, just do it for other partition.
      <code>
      ...
      gmirror label root /dev/gpt/root0 /dev/gpt/root1
      gmirror label var /dev/gpt/var0 /dev/gpt/var1
      gmirror label tmp /dev/gpt/var0 /dev/gpt/tmp1
      gmirror label usr/dev/gpt/var0 /dev/gpt/usr1
      ...
      </code>
      
      Do newfs then mount everything to /mnt
      
      <code>
      ...
      ...
      newfs -U -L root /dev/mirror/root
      newfs -U -L var /dev/mirror/var
      newfs -U -L tmp /dev/mirror/tmp
      newfs -U -L usr /dev/mirror/usr
      ...
      ...

      mount /dev/mirror/root /mnt
      mkdir /mnt/var
      mount /dev/mirror/var /mnt/var
      mkdir /mnt/tmp
      mount /dev/mirror/tmp /mnt/tmp
      mkdir /mnt/usr
      mount /dev/mirror/usr /mnt/usr

      And just type “exit”, it will resume setup operation, then just follow Adam’s instruction.

      Thanks Adam for this post…

          1. rizza
            i just follow intruction from Krissada to partition var,tmp,usr and now i found new problem.
            when i reboot without dvd ,its written

            gptboot: no /boot/loader on 0:ad(0p3)
            gptboot: no /boot/kernel/kernel on 0:ad(0p3)

            can you solve my case as fast as you can?
            please help me to solve my problem sir.

    3. koorosh
      Hi
      can you create guide for raid 10 too !?
      i’m little confused about raid 10 , but your step by step guide is very helpful.
      1. Adam Strohl Post author
        Thank you, but I would not recommend striped + mirrored ZFS as it has less redundancy than a RAIDZ2 would. Any two disks could fail and the RAID would continue working with RAIDZ2. With striped and mirrored vdevs (“RAID10” with ZFS) if you lose two disk within the same stripe your array will fail.
    4. Pingback: Geom Root mirror & ZFS Data | nq1320
    5. Sigmar Wiesmayr
      Adam,

      thanks for this great instructions.

      A detail, perhaps I am wrong, but at step 6, shouldn’t it be
      echo ‘geom_mirror_load=”YES”‘ >> /mnt/boot/loader.conf and
      echo ‘daily_status_gmirror_enable=”YES”‘ >> /mnt/etc/periodic.conf

      Regards,
      Sigmar

      1. Adam Strohl Post author
        Hey Sigmar, the instructions in step 6 are correct as the “final changes” shell chroots to the new filesystem. So all paths should not have /mnt in front of them at that step.
    6. Pingback: Geom Root mirror & ZFS Data | Techy Stuff

    Leave a Reply

    Your email address will not be published. Required fields are marked *

Contact Us

 

 

Share your vote!


Do you like this post?
  • Fascinated
  • Happy
  • Sad
  • Angry
  • Bored
  • Afraid

Leave a Reply

Your email address will not be published. Required fields are marked *


Please help to input verification code