Virtual Architects Support Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Napp-IT ZFS Storage Server ======

//**As the saying goes: "With RAID, you can have performance, capacity, and integrity - pick any two."**//

See also **[[computing:storage:zfs|ZFS]]**

See also **[[computing:storage:zfs_snapshot|ZFS Snapshots and Napp-IT Replication]]**

See also **[[https://www.napp-it.org/doc/downloads/flash_lsi_sas.pdf|Flash LSI IR to IT Mode]]**

**License Swap**: http://napp-it.org/extensions/swap_en.html 

**Official Handbook**: http://www.napp-it.org/doc/downloads/napp-it.pdf

**Mini-HOWTO**: ftp://ftp.uni-duisburg.de/Solaris/napp-it/napp-it.pdf

**All-in-One**: http://www.napp-it.org/doc/downloads/all-in-one.pdf

**Best HBAs**: https://www.servethehome.com/buyers-guides/top-hardware-components-napp-omnios-nas-servers/top-picks-for-napp-it-and-omnios-hbas-host-bus-adapters/

**Best Hardware**: https://www.servethehome.com/buyers-guides/top-hardware-components-napp-omnios-nas-servers/

https://forums.servethehome.com/index.php?threads/performance-tuning-three-monster-zfs-systems.3921/

http://www.overclockers.com/forums/showthread.php?t=705879

http://researchblogs.cs.bham.ac.uk/specialist-computing/2012/05/building-an-openindiana-based-zfs-file-server/

Supported sharing protocols:

    * AFP
    * CIFS/SMB
    * NFS
    * iSCSI
    * FTP
    * rsync
    * FOE

===== OmniOS =====

See also **[[computing:unix:omnios|OmniOS]]**

See also **[[http://www.napp-it.org/downloads/omnios_en.html|Napp-IT on OmniOS]]**

  * Recommended host OS for Napp-IT
  * Networking not configured during installation

==== Configure the Primary (Management) Network Interface ====

<note warning>**Don't mess with networking** in the Napp-IT web interface unless you are prepared to manually fix things that go wrong using the CLI!</note>

:!: Other interfaces, hostname, domain name and DNS servers can be configured with Napp-IT after it's installed, including MTU for Jumbo Frames.

**System -> Network**

Disable ''nwam'' (Network AutoMagic):

<file>
svcs nwam
svcs network/physical:default

svcadm disable svc:/network/physical:nwam
svcadm enable svc:/network/physical:default
</file>

<file>
dladm show-phys                                     # show existing
dladm show-link                                     # show available
ipadm show-if                                       # show existing IP configurations

ipadm create-if igb0                                # create the IF
ipadm create-addr -T static -a 192.168.1.4 igb0/v4  # set a static IP

route -p add default 192.168.1.1                    # set default route to your gateway

echo 'nameserver 8.8.8.8' >> /etc/resolv.conf       # or edit resolv.conf as needed

echo 'yourdomainname'  >> /etc/resolv.conf          # or edit resolv.conf as needed

echo 'yourdomainname' > /etc/defaultdomain           # used by NIS

cp /etc/nsswitch.dns /etc/nsswitch.conf             # use DNS for name resolution

dig google.com                                      # to test

reboot
</file>

===== Install Napp-IT =====

  * Install OmniOS normally from CD or USB flash drive
  * Update OmniOS per the OmniOS wiki page before installing Napp-IT

As root:

<file>
wget -O - www.napp-it.org/nappit | perl
</file>

:!: The Napp-IT install changes the root password so we need to change it back.

Before rebooting type:

<file>
passwd root                          # reset the root password to set SMB password
passwd napp-it                       # for the Napp-IT web interface 'admin' user
reboot
</file>

==== Updating to the Latest Napp-IT Version ====

<file>
wget -O - www.napp-it.org/nappit | perl
</file>

  * Reboot
  * Re-enter the root password via ''passwd root''
  * Check and/or recreate your jobs due to a new job management with editable timer and job-settings
  * Re-enable auto-service

==== Napp-IT Configuration ====

http://napp-it.org/doc/downloads/napp-it.pdf

:!: Napp-IT should only be used on secure networks.  SSL is not available by default on the web interface.

:!: Delete '' /var/web-gui/_log/napp-it.cfg'' to reset to defaults.

Using a web browser:

<file>
http://<server ip address>:81
https://<server ip address>:82
</file>

=== First Login ===

**admin**, **no password**

:!: Change the password now.

=== Configure Networking ===

Also configure hostname, DNS servers, domain name, etc.:

**System -> Network -> IP Address**

=== Configure E-Mail ===

:!: Typically Napp-IT is used in a secure network.  However, if your mail server is external, you will want to configure TLS mail encryption.

<file>
perl -MCPAN -e shell

notest install Net::SSLeay
notest install IO::Socket::SSL
notest install Net::SMTP::TLS
exit
</file>

Enable and test SMTP mail without TLS encryption:

**About -> Settings**

**Jobs -> E-mail -> SMTP Test**

Once standard SMTP mail works, configure TLS (encrypted connections):

**Jobs -> TLS E-mail -> Enable TLS Mail**

**Jobs -> TLS E-mail -> TLS Test**

=== Enable Services ===

  * Auto-Service
    * This is the master service required for other scheduled jobs
      * **Jobs -> Auto-Service -> Enable auto 15min**
  * Scrubs
    * Create a scrub job for each pool you create
      * **Jobs -> Scrub -> Create auto scrub job**
  * Create jobs for e-mailed status and alerts
      * **Jobs -> E-mail -> Status**
      * **Jobs -> E-mail -> Alert**

=== Configure Storage ===

http://www.overclockers.com/forums/showthread.php?t=705879

:!: Here we create a pool of mirrored vdevs (RAID10).

Create a pool with the first vdev being a pair of mirrored drives:

**Pools -> Create Pool**

  * Name the pool
  * Choose the first two disks to be mirrored and choose **Mirror**
  * **Submit**

Add more mirrors to this pool until done:

**Pools -> Extend Pool**

  * Select the proper pool
  * Select two more disks and choose **Mirror**
  * **Submit**

Add L2ARC or ZIL (per pool):

**Pools -> Extend Pool**

  * Select the proper pool
  * Select the SSD drive you want to use
  * Choose the ''read-cache'' or ''write-log'' function
  * **Submit**

=== iSCSI ===

http://blog.allanglesit.com/2012/02/adventures-in-zfs-configuring-iscsi-targets/

http://docs.oracle.com/cd/E19963-01/html/821-1459/fncpi.html

http://blogs.citrix.com/2011/06/01/sizing-luns-a-citrix-perspective/

  * VMs per LUN = Approx 20 max
  * iSCSI provides block storage
  * Host Groups - which initiators/clients can see the LU
  * Target Groups - which targets will be able to provide access to the LU

== Example Naming Conventions ==

^iSCSI Device        ^Name                               ^Example                                 ^
|LU                  |lu-name                            |lu-vhd-infra                            |
|Target Name         |iqn.2010-09.org.openindiana:t-name |iqn.2010-09.org.openindiana:t-vhd-infra |
|Target Alias        |t-name                             |t-vhd-infra                             |
|Target Group        |tg-name                            |tg-vhd-infra                            |
|Host Group          |hg-name                            |hg-vhd-infra                            |

== Setup a Running Configuration ==

http://docs.oracle.com/cd/E23824_01/html/821-1459/fnnop.html

:!: Verify that the Comstar and iSCSI services are running first:

**Services -> Comstar**

Once we verify the services are running, we configure a LU for use:

  - You must **define a logical unit**
    - **Comstar -> Logical Units -> Create Thin Prov LU**
    - File-based thin provisioned LU's are suggested
      * You can copy/move virtual hdd files
    - Disable Writeback Cache on your LU if you prefer data-security with sync-writes
      * Lowers performance unless you have a fast ZIL drive
  - You must **define a target**
    - **Comstar -> Targets -> Create iSCSI Target**
    - The target is what clients connect to
    - A target can contain multiple logical units
  - You must **define a target-group** and **add your target(s) as a member**
    - **Comstar -> Target Groups -> Create Target Group**
    - **Comstar -> Target Groups -> Add Members**
    - Views cannot be set to targets, only to target groups
  - You must **set a view** for each logical unit to a target-group or the LU remains invisible
    - **Comstar -> Views - Add a View**
      * Set the target group to ''all'' for visibility to all clients

===== Tuning =====

http://www.napp-it.org/manuals/tuning.html

Disable ''atime'':

<file>
zfs set atime=off <poolname>
</file>

===== Useful Commands =====

http://docs.oracle.com/cd/E19963-01/html/821-1459/fncpi.html

LUN:

<file>
svcs stmf
stmfadm list-lu
stmfadm list-view -l <lu-name>
</file>

Target:

<file>
svcs -l iscsi/target

/var/svc/log/network-iscsi-target:default.log

itadm list-target -v
</file>

===== S3 =====

http://www.napp-it.org/doc/downloads/cloudsync.pdf

https://forums.servethehome.com/index.php?threads/amazon-s3-compatible-zfs-cloud-with-minio.27524/

  * Update ''napp-it'' (**About -> Update**)
  * Use menu **Services -> minIO S3 Services** to install ''minIO''
  * Use menu **ZFS Filesystems -> S3cloud -> unset** to activate (ex: on port 9000)

You can share the same filesystem via SMB and S3, but there is no file locking.

Via SMB, you will find a folder ''S3_data'' and ''S3_config'' with S3 data.

Open a browser (or any S3 client) with address ip:9000.

===== NFS =====

**Mini-HOWTO**: ftp://ftp.uni-duisburg.de/Solaris/napp-it/napp-it.pdf

**NFS Share Permissions**: http://forums.freebsd.org/showthread.php?t=26801

**More NFS Share Permissions**: http://forums.freebsd.org/showthread.php?t=26801

**Using NFS for VMware ESX**: http://mordtech.com/2008/12/11/leveraging-zfs-nfs-for-esxi/

  - Create a data pool to store your files on
  - Create ZFS Filesystem (dataset) on your pool
  - Set ACL permissions of pool/dataset to something like
    - ''root=full'' with inheritance on for newly created files and folders
    - ''everyone@=full'' or ''everyone@=modify'' with inheritance on for newly created files and folders
  - Share the dataset via NFS
    - For virtualization hosts, you may need to specify options when enabling NFS sharing:
      - <file>sharenfs=root=ip.of.virt.host</file>
  - Use the new NFS share with a URL like:
    - <file>hostnameorip:/pool/dataset</file>

{{ :computing:storage:napp-it_zfs_folder_acl.png?direct&750 |ZFS Folder ACLs}}

{{ :computing:storage:napp-it_zfs_smb_acl.png?direct&750 |ZFS Share ACLs}}


===== CIFS/SMB =====

:!: You have to restart the SMB service after making changes!

==== HowTo ====

**Mini-HOWTO**: ftp://ftp.uni-duisburg.de/Solaris/napp-it/napp-it.pdf

  - Create/Add at least one SMB-user (not root) to smb-group administrators
  - Set all needed ACLs from Windows (right-click on folder -> Properties -> Security)
  - If you smb-share a ZFS-folder via napp-it, these settings are used:
    - Set unix permissions to 777 (or some ACL and share options will not work)
    - Set the folder ACL from "nearly everything is denied" to the following ACL:
      - root=full access
      - everybody@=modify
  - Set share-level ACL‘s to everybody=full access
  - See the linked PDF above for various permissions and ACL settings
    - You mostly want to connect from Windows and manage ACLs/Permissions from there

==== Folder ACLs ====

**ZFS Filesystems -> ACL Extension -> ACL on Folders -> <SMB Share>**

Remove all ACLs:

<file>
/usr/bin/chmod -R A- /pool/share
</file>

Add ACLs (recursively) for SMB shares (order is important - ''root'' rule will be number ''0''):

<file>
/usr/bin/chmod -R A=everyone@:modify_set:file_inherit/dir_inherit:allow /pool/share
/usr/bin/chmod -R A+user:root:full_set:file_inherit/dir_inherit:allow /pool/share
</file>

{{ :computing:storage:acl_on_folders.png?direct&800 |Folder ACLs on SMB Shares}}

==== CIFS Troubleshooting ====

  - Don't use numeric (octal) ''chmod'' commands on SMB shares or the ACLs get messed up
  - Set Unix permissions to ''777'' on top level (share) folder
  - Reset the ''root'' password with the ''passwd'' command after enabling CIFS
    * This sets the ''root'' SMB user password too
  - In Windows, log in as the ''root'' user to set permissions, if necessary

===== Block or File Based Storage =====

See also: **[[computing:storage:iscsi_vs_nfs|iSCSI vs. NFS]]**

http://searchservervirtualization.techtarget.com/tip/ISCSI-vs-NFS-for-virtualization-shared-storage

http://www.brentozar.com/archive/2012/05/storage-protocol-basics-iscsi-nfs-fibre-channel-fcoe/

**NFS or iSCSI?**

:!: The bottom line is that iSCSI may perform better but will be more trouble to manage.

  * File (NFS) or block (iSCSI) storage
  * Proxmox (OpenVZ) Containers must be on NFS (or local storage)
  * VMware ESXi needs NFS

===== Disk Errors =====

**Napp-IT -> System -> Basic Statistics -> Disks**

Every five seconds:

<file>
iostat 5
</file>

Extended disk statistics:

<file>
iostat -xtc
iostat -xn 1 2               # use second value only
</file>

**Soft/Hard/Transfer errors** are warnings from ''iostat''.  Look at them as warnings not real errors like ZFS checksum errors, but they indicate problems.

Look also at the **wait value on writes (%w)**.  If you have significantly high wait or busy values on a single disk this indicates a problem as well.

{{ :computing:storage:napp-it_zfs_diskinfos.png?750 |ZFS Disk Errors}}

==== Replace a Failed Disk ====

In the web GUI:

**Disks -> Replace**

  * Select the failed drive
  * Select the new drive you already installed
  * Click 'Replace'

If you simply replace a failed boot drive with another drive (system off), you can then:

<file>
zpool status rpool
zpool replace rpool device
</file>

==== rpool ====

https://omnios.org/info/migrate_rpool

Install the boot block and boot loader of mirrored boot drives:

<file>
bootadm install-bootloader -P rpool
</file>

:!: Test or switch boot drives in the system BIOS.

===== File Manager =====

Use Midnight Commander with the command ''mc''.

  * Works well with PuTTY
  * May have issues at console with function keys or character sets.

===== Troubleshooting =====

In case of problems, use the console and check:

  * poolstatus: ''zpool status''
  * zfsstatus: ''zfs list''
  * controller: ''cfgadm -avl''
  * diskstatus: ''format'' or ''iostat -En''
  * services: ''prstat''
  * napp-it webserver restart: ''/etc/init.d/napp-it restart''
  * or try a reboot or reinstall via ''wget''

===== SFTP =====

  - Add SFTP user to SSH ''AllowUsers'' or ''AllowGroups''
    - ''AllowUsers root backupuser''
  - Modify ''/etc/passwd'' SFTP user home directory and restricted shell
    * ''/bin/rbash''
  - Restart SSH service
    * ''svcadm restart svc:/network/ssh:default''

===== FTP =====

http://www.napp-it.org/extensions/proftpd_en.html

http://www.proftpd.org/docs/howto/

:!: Note that **FTP is an add-on** not included in the base Napp-IT installation.

:!: Try **Passive Mode** to connect if problems.

==== Install ProFTPd Add-On ====

<file>
wget -O - www.napp-it.org/proftpd | perl
</file>

This starts the service and can also be  used to restart the ProFTPd service after making configuration changes:

**Services -> FTP -> Enable proftpd**

<file>
svcs proftpd
svcadm enable proftpd
svcadm disable proftpd
</file>

==== Add a User for FTP Access ====

**User -> Add Local User**

:!: No home directory is created.

:!: Users must be deleted with the ''userdel'' command at the CLI.

==== Virtual Server Configuration ====

**ZFS Filesystems -> FTP (link per filesystem)**

:!: You must manually define the Virtual Server by clicking the **set other manually** link.

:!: You can have only one Virtual Server defined per port (i.e port 21).

<file>
<VirtualHost 10.1.1.9>
  ServerName                     "FTP Server"
  Port                           21
  Umask                          022

  <Limit LOGIN>
    DenyAll
  </Limit>

  <Anonymous /backup/mybackup>
    User                         mybackupuser
    Group                        nobody
    AnonRequirePassword          on
    AllowOverwrite               on

    <Limit LOGIN>
      AllowUser                  mybackupuser
      DenyAll
    </Limit>

    <Limit WRITE>
      AllowUser                  mybackupuser
      DenyAll
    </Limit>
  </Anonymous>
</VirtualHost>
</file>

===== Notes From Others =====

Mostly stolen from https://hardforum.com where the user ''_Gea'' is the primary developer of Napp-IT.

**On link aggregation:**

Link aggregation may help a little with a lot of parallel connections
but mostly its complicated only with a lot of possible problems and
no or minimal performance advantage.

i always follow these rules

1. keep it simple
2. use 10 Gb/ FC if you need speed
3. If you have an All-In-One ESXi/ SAN solution, use one VLAN Uplink
from ESXi to your physical switch (1 GB or 10 Gbit) and divide your LANS there
Use highspeed virtual vnics to internally connect your VM's with your SAN
4, On ESXi use virtual software switches and not physical nics beside failover

1 Gb aggregation is outdated. 10 GB is on the way to become cheap.
Currently 2x10 Gb cards are about 300 Euro but you can expect them to
be onboard in 2012 on better mainboards or as cheap as good 1 GB Nics 5 years ago

10 Gb on switches is currently availabe for about 250 Euro per port.
I use HP 2910 switches with up to 4 x 10 Gb ports. They are not really cheap
(about 1300 Euro with 24 x 1Gb Ports + 2 x 10 Gb for about 500 Euro)
but affordable if you need the speed.

If you only need high-speed between one server and a few clients, you do not need
a switch immideatly (example small video editing pool) and can connect them directly
and buy the 10 Gb switch later.

Gea 

**More on link aggregation:**

 with link aggregation, you complicate things with often no or minimal benefit and add an extra
problem field for example together with jumbo frames (mostly not working at all)
in my opinion, its not worth the effort today

about your pools:
if you need best IO and speed for VM's use always mirrors so you pool 1 is perfect
- add at least one hotspare!!
- you may add a ssd read cache and eventually a mirrored write cache (Hybrid storage)


pool2
a hotfix to a Raid-Z is not very efficient.
if you have a failure you need a rebuild with a at this moment untested disk
use next Raid-Z level Raid-Z2 instead and you have a 'hot' hotfix
use hotfix always on mirrors and on a Raid-Z3 if needed

Depending on your workload, SSD cache dtives can help to improve performance.
For my own i switched to SSD only pools as ESXi datastore.
(although they are not as reliable as good SAS disks, so i use 3 x mirrors now)

The time for expensive 15k SAS is over for new installations (imho)


Gea 

**On networking:**

 I have problems with Solaris 11 and Link Aggression, I have a dell powerconnect switch, I put it to LAG mode and set up LACP at solaris, I set a static IP and don't get any connection with the local network, I check, dladm say it is up and I put it to DHCP and I get a IP from the router but still no connection with the local network, WHATS UP?
Reply With Quote
  #1147  
Old 07-04-2011, 11:07 AM
ChrisBenn Limp Gawd, 1.6 Years
	  	
Status: ChrisBenn is offline  
The dell powerconnect switches require a L2 policy and a static aggregation (no LACP)

if you have 2 interfaces, say ige0 and ige1

<file>
ipadm delete-if ige0
ipadm delete-if ige1
dladm create-aggr -P L2 -l ige0 -l ige1 aggr1
ipadm create-addr -T dhcp aggr1/v4
</file>

should work for you. This is assuming you have disabled the nwam service and are using the physical service:


<file>
svcadm disable svc:/network/physical:nwam
svcadm enable svc:/network/physical:default
</file> 

**On disk failures:**

So if you have a bunch of disks in a ZFS pool how can you move them to a new server?
If you have done it like suggested (Use HBA controller, never use hardware-raid)
you can just plug your disks into your new computer with any disk controller
and import your pool - no problem -

Quote:
How does rebuilding work when you lose a drive?
If you have set ZFS pool property autoreplace=on you just need to replace
a failed drive, otherwise plug in a new disk and do a replace failled drive -> new disk

If your controller does not support hot-plug you need a reboot after plug-in new disks

Gea 

**On SMB file sharing:**

Hello, I run Solaris 11 Express and napp-it for stuff liek SMB and zpools

I was wondering how to I make custom SMB users or groups, like users that only can access some files, they can read at one place and write on another, how does this work?
With napp-it you can create user and smb-groups in menu user.
Connect from Windows as user root and set desired file and folder ACL
(works from Win XP pro, Win 2003 and Win 7 pro, problems are reported with home editions
and Win 7 ultimate)

Problem: Solaris ACL are order sensitive - Windows ACL not
non-trivial ACL should be set from Solaris

From Solaris you can set ACL via CLI or via napp-it ACL extension
(in development, currently you can set share level ACL and ACL on shared folders
not on other files and folders)


Gea 


Exactly what Gea said, but to give you specific commands

#zpool import
to list all the pools found

#zpool import poolname
to import the pool.

If you had a failure and didn't do a clean export you just need to use the -f (force) switch
#zpool import -f poolname 

**About overflow protection:**

Its always bad, if you fill up your pool up to 100% with data.
Overflow protection sets a 10% reservation on the pool ZFS itself.

The available space for your ZFS folder is then 90%.
That means, if you fill your folder up to the max, 10% of the pool remains always free.
You can check/modify that always in menu ZFS folder under reservation.

Gea
Virtual Architects Support Wiki

User Tools

Site Tools

Page Tools
