As the saying goes: “With RAID, you can have performance, capacity, and integrity - pick any two.”

See also ZFS

See also ZFS Snapshots and Napp-IT Replication

See also Flash LSI IR to IT Mode

License Swap: http://napp-it.org/extensions/swap_en.html

Official Handbook: http://www.napp-it.org/doc/downloads/napp-it.pdf

Mini-HOWTO: ftp://ftp.uni-duisburg.de/Solaris/napp-it/napp-it.pdf

All-in-One: http://www.napp-it.org/doc/downloads/all-in-one.pdf

Best HBAs: https://www.servethehome.com/buyers-guides/top-hardware-components-napp-omnios-nas-servers/top-picks-for-napp-it-and-omnios-hbas-host-bus-adapters/

Best Hardware: https://www.servethehome.com/buyers-guides/top-hardware-components-napp-omnios-nas-servers/

https://forums.servethehome.com/index.php?threads/performance-tuning-three-monster-zfs-systems.3921/

http://www.overclockers.com/forums/showthread.php?t=705879

http://researchblogs.cs.bham.ac.uk/specialist-computing/2012/05/building-an-openindiana-based-zfs-file-server/

Supported sharing protocols:

• AFP
• CIFS/SMB
• NFS
• iSCSI
• FTP
• rsync
• FOE

See also OmniOS

See also Napp-IT on OmniOS

• Recommended host OS for Napp-IT
• Networking not configured during installation

Other interfaces, hostname, domain name and DNS servers can be configured with Napp-IT after it's installed, including MTU for Jumbo Frames.

System → Network

Disable nwam (Network AutoMagic):

svcs nwam
svcs network/physical:default

svcadm disable svc:/network/physical:nwam
svcadm enable svc:/network/physical:default

dladm show-phys                                     # show existing
dladm show-link                                     # show available
ipadm show-if                                       # show existing IP configurations

ipadm create-if igb0                                # create the IF
ipadm create-addr -T static -a 192.168.1.4 igb0/v4  # set a static IP

route -p add default 192.168.1.1                    # set default route to your gateway

echo 'nameserver 8.8.8.8' >> /etc/resolv.conf       # or edit resolv.conf as needed

echo 'yourdomainname'  >> /etc/resolv.conf          # or edit resolv.conf as needed

echo 'yourdomainname' > /etc/defaultdomain           # used by NIS

cp /etc/nsswitch.dns /etc/nsswitch.conf             # use DNS for name resolution

dig google.com                                      # to test

reboot

• Install OmniOS normally from CD or USB flash drive
• Update OmniOS per the OmniOS wiki page before installing Napp-IT

As root:

wget -O - www.napp-it.org/nappit | perl

The Napp-IT install changes the root password so we need to change it back.

Before rebooting type:

passwd root                          # reset the root password to set SMB password
passwd napp-it                       # for the Napp-IT web interface 'admin' user
reboot

wget -O - www.napp-it.org/nappit | perl

• Reboot
• Re-enter the root password via passwd root
• Check and/or recreate your jobs due to a new job management with editable timer and job-settings
• Re-enable auto-service

http://napp-it.org/doc/downloads/napp-it.pdf

Napp-IT should only be used on secure networks. SSL is not available by default on the web interface.

Delete /var/web-gui/_log/napp-it.cfg to reset to defaults.

Using a web browser:

http://<server ip address>:81
https://<server ip address>:82

admin, no password

Change the password now.

Also configure hostname, DNS servers, domain name, etc.:

System → Network → IP Address

Typically Napp-IT is used in a secure network. However, if your mail server is external, you will want to configure TLS mail encryption.

perl -MCPAN -e shell

notest install Net::SSLeay
notest install IO::Socket::SSL
notest install Net::SMTP::TLS
exit

Enable and test SMTP mail without TLS encryption:

About → Settings

Jobs → E-mail → SMTP Test

Once standard SMTP mail works, configure TLS (encrypted connections):

Jobs → TLS E-mail → Enable TLS Mail

Jobs → TLS E-mail → TLS Test

• Auto-Service
  • This is the master service required for other scheduled jobs
    • Jobs → Auto-Service → Enable auto 15min
• Scrubs
  • Create a scrub job for each pool you create
    • Jobs → Scrub → Create auto scrub job
• Create jobs for e-mailed status and alerts
  • Jobs → E-mail → Status
  • Jobs → E-mail → Alert

http://www.overclockers.com/forums/showthread.php?t=705879

Here we create a pool of mirrored vdevs (RAID10).

Create a pool with the first vdev being a pair of mirrored drives:

Pools → Create Pool

• Name the pool
• Choose the first two disks to be mirrored and choose Mirror
• Submit

Add more mirrors to this pool until done:

Pools → Extend Pool

• Select the proper pool
• Select two more disks and choose Mirror
• Submit

Add L2ARC or ZIL (per pool):

Pools → Extend Pool

• Select the proper pool
• Select the SSD drive you want to use
• Choose the read-cache or write-log function
• Submit

http://blog.allanglesit.com/2012/02/adventures-in-zfs-configuring-iscsi-targets/

http://docs.oracle.com/cd/E19963-01/html/821-1459/fncpi.html

http://blogs.citrix.com/2011/06/01/sizing-luns-a-citrix-perspective/

• VMs per LUN = Approx 20 max
• iSCSI provides block storage
• Host Groups - which initiators/clients can see the LU
• Target Groups - which targets will be able to provide access to the LU

http://docs.oracle.com/cd/E23824_01/html/821-1459/fnnop.html

Verify that the Comstar and iSCSI services are running first:

Services → Comstar

Once we verify the services are running, we configure a LU for use:

1. You must define a logical unit
   1. Comstar → Logical Units → Create Thin Prov LU
   2. File-based thin provisioned LU's are suggested
      • You can copy/move virtual hdd files
   3. Disable Writeback Cache on your LU if you prefer data-security with sync-writes
      • Lowers performance unless you have a fast ZIL drive
2. You must define a target
   1. Comstar → Targets → Create iSCSI Target
   2. The target is what clients connect to
   3. A target can contain multiple logical units
3. You must define a target-group and add your target(s) as a member
   1. Comstar → Target Groups → Create Target Group
   2. Comstar → Target Groups → Add Members
   3. Views cannot be set to targets, only to target groups
4. You must set a view for each logical unit to a target-group or the LU remains invisible
   1. Comstar → Views - Add a View
      • Set the target group to all for visibility to all clients

http://www.napp-it.org/manuals/tuning.html

Disable atime:

zfs set atime=off <poolname>

http://docs.oracle.com/cd/E19963-01/html/821-1459/fncpi.html

LUN:

svcs stmf
stmfadm list-lu
stmfadm list-view -l <lu-name>

Target:

svcs -l iscsi/target

/var/svc/log/network-iscsi-target:default.log

itadm list-target -v

http://www.napp-it.org/doc/downloads/cloudsync.pdf

https://forums.servethehome.com/index.php?threads/amazon-s3-compatible-zfs-cloud-with-minio.27524/

• Update napp-it (About → Update)
• Use menu Services → minIO S3 Services to install minIO
• Use menu ZFS Filesystems → S3cloud → unset to activate (ex: on port 9000)

You can share the same filesystem via SMB and S3, but there is no file locking.

Via SMB, you will find a folder S3_data and S3_config with S3 data.

Open a browser (or any S3 client) with address ip:9000.

Mini-HOWTO: ftp://ftp.uni-duisburg.de/Solaris/napp-it/napp-it.pdf

NFS Share Permissions: http://forums.freebsd.org/showthread.php?t=26801

More NFS Share Permissions: http://forums.freebsd.org/showthread.php?t=26801

Using NFS for VMware ESX: http://mordtech.com/2008/12/11/leveraging-zfs-nfs-for-esxi/

1. Create a data pool to store your files on
2. Create ZFS Filesystem (dataset) on your pool
3. Set ACL permissions of pool/dataset to something like
   1. root=full with inheritance on for newly created files and folders
   2. everyone@=full or everyone@=modify with inheritance on for newly created files and folders
4. Share the dataset via NFS
   1. For virtualization hosts, you may need to specify options when enabling NFS sharing:

      1. sharenfs=root=ip.of.virt.host

5. Use the new NFS share with a URL like:

   1. hostnameorip:/pool/dataset

You have to restart the SMB service after making changes!

Mini-HOWTO: ftp://ftp.uni-duisburg.de/Solaris/napp-it/napp-it.pdf

1. Create/Add at least one SMB-user (not root) to smb-group administrators
2. Set all needed ACLs from Windows (right-click on folder → Properties → Security)
3. If you smb-share a ZFS-folder via napp-it, these settings are used:
   1. Set unix permissions to 777 (or some ACL and share options will not work)
   2. Set the folder ACL from “nearly everything is denied” to the following ACL:
      1. root=full access
      2. everybody@=modify
4. Set share-level ACL‘s to everybody=full access
5. See the linked PDF above for various permissions and ACL settings
   1. You mostly want to connect from Windows and manage ACLs/Permissions from there

ZFS Filesystems → ACL Extension → ACL on Folders → <SMB Share>

Remove all ACLs:

/usr/bin/chmod -R A- /pool/share

Add ACLs (recursively) for SMB shares (order is important - root rule will be number 0):

/usr/bin/chmod -R A=everyone@:modify_set:file_inherit/dir_inherit:allow /pool/share
/usr/bin/chmod -R A+user:root:full_set:file_inherit/dir_inherit:allow /pool/share

1. Don't use numeric (octal) chmod commands on SMB shares or the ACLs get messed up
2. Set Unix permissions to 777 on top level (share) folder
3. Reset the root password with the passwd command after enabling CIFS
   • This sets the root SMB user password too
4. In Windows, log in as the root user to set permissions, if necessary

See also: iSCSI vs. NFS

http://searchservervirtualization.techtarget.com/tip/ISCSI-vs-NFS-for-virtualization-shared-storage

http://www.brentozar.com/archive/2012/05/storage-protocol-basics-iscsi-nfs-fibre-channel-fcoe/

NFS or iSCSI?

The bottom line is that iSCSI may perform better but will be more trouble to manage.

• File (NFS) or block (iSCSI) storage
• Proxmox (OpenVZ) Containers must be on NFS (or local storage)
• VMware ESXi needs NFS

Napp-IT → System → Basic Statistics → Disks

Every five seconds:

iostat 5

Extended disk statistics:

iostat -xtc
iostat -xn 1 2               # use second value only

Soft/Hard/Transfer errors are warnings from iostat. Look at them as warnings not real errors like ZFS checksum errors, but they indicate problems.

Look also at the wait value on writes (%w). If you have significantly high wait or busy values on a single disk this indicates a problem as well.

In the web GUI:

Disks → Replace

• Select the failed drive
• Select the new drive you already installed
• Click 'Replace'

Use Midnight Commander with the command mc.

• Works well with PuTTY
• May have issues at console with function keys or character sets.

In case of problems, use the console and check:

• poolstatus: zpool status
• zfsstatus: zfs list
• controller: cfgadm -avl
• diskstatus: format or iostat -En
• services: prstat
• napp-it webserver restart: /etc/init.d/napp-it restart
• or try a reboot or reinstall via wget

1. Add SFTP user to SSH AllowUsers or AllowGroups
   1. AllowUsers root backupuser
2. Modify /etc/passwd SFTP user home directory and restricted shell
   • /bin/rbash
3. Restart SSH service
   • svcadm restart svc:/network/ssh:default

http://www.napp-it.org/extensions/proftpd_en.html

http://www.proftpd.org/docs/howto/

Note that FTP is an add-on not included in the base Napp-IT installation.

Try Passive Mode to connect if problems.

wget -O - www.napp-it.org/proftpd | perl

This starts the service and can also be used to restart the ProFTPd service after making configuration changes:

Services → FTP → Enable proftpd

svcs proftpd
svcadm enable proftpd
svcadm disable proftpd

User → Add Local User

No home directory is created.

Users must be deleted with the userdel command at the CLI.

ZFS Filesystems → FTP (link per filesystem)

You must manually define the Virtual Server by clicking the set other manually link.

You can have only one Virtual Server defined per port (i.e port 21).

<VirtualHost 10.1.1.9>
  ServerName                     "FTP Server"
  Port                           21
  Umask                          022

  <Limit LOGIN>
    DenyAll
  </Limit>

  <Anonymous /backup/mybackup>
    User                         mybackupuser
    Group                        nobody
    AnonRequirePassword          on
    AllowOverwrite               on

    <Limit LOGIN>
      AllowUser                  mybackupuser
      DenyAll
    </Limit>

    <Limit WRITE>
      AllowUser                  mybackupuser
      DenyAll
    </Limit>
  </Anonymous>
</VirtualHost>

Mostly stolen from https://hardforum.com where the user _Gea is the primary developer of Napp-IT.

On link aggregation:

Link aggregation may help a little with a lot of parallel connections but mostly its complicated only with a lot of possible problems and no or minimal performance advantage.

i always follow these rules

1. keep it simple 2. use 10 Gb/ FC if you need speed 3. If you have an All-In-One ESXi/ SAN solution, use one VLAN Uplink from ESXi to your physical switch (1 GB or 10 Gbit) and divide your LANS there Use highspeed virtual vnics to internally connect your VM's with your SAN 4, On ESXi use virtual software switches and not physical nics beside failover

1 Gb aggregation is outdated. 10 GB is on the way to become cheap. Currently 2×10 Gb cards are about 300 Euro but you can expect them to be onboard in 2012 on better mainboards or as cheap as good 1 GB Nics 5 years ago

10 Gb on switches is currently availabe for about 250 Euro per port. I use HP 2910 switches with up to 4 x 10 Gb ports. They are not really cheap (about 1300 Euro with 24 x 1Gb Ports + 2 x 10 Gb for about 500 Euro) but affordable if you need the speed.

If you only need high-speed between one server and a few clients, you do not need a switch immideatly (example small video editing pool) and can connect them directly and buy the 10 Gb switch later.

Gea

More on link aggregation:

with link aggregation, you complicate things with often no or minimal benefit and add an extra problem field for example together with jumbo frames (mostly not working at all) in my opinion, its not worth the effort today

about your pools: if you need best IO and speed for VM's use always mirrors so you pool 1 is perfect - add at least one hotspare!! - you may add a ssd read cache and eventually a mirrored write cache (Hybrid storage)

pool2 a hotfix to a Raid-Z is not very efficient. if you have a failure you need a rebuild with a at this moment untested disk use next Raid-Z level Raid-Z2 instead and you have a 'hot' hotfix use hotfix always on mirrors and on a Raid-Z3 if needed

Depending on your workload, SSD cache dtives can help to improve performance. For my own i switched to SSD only pools as ESXi datastore. (although they are not as reliable as good SAS disks, so i use 3 x mirrors now)

The time for expensive 15k SAS is over for new installations (imho)

Gea

On networking:

I have problems with Solaris 11 and Link Aggression, I have a dell powerconnect switch, I put it to LAG mode and set up LACP at solaris, I set a static IP and don't get any connection with the local network, I check, dladm say it is up and I put it to DHCP and I get a IP from the router but still no connection with the local network, WHATS UP? Reply With Quote

#1147  

Old 07-04-2011, 11:07 AM ChrisBenn Limp Gawd, 1.6 Years

Status: ChrisBenn is offline The dell powerconnect switches require a L2 policy and a static aggregation (no LACP)

if you have 2 interfaces, say ige0 and ige1

ipadm delete-if ige0
ipadm delete-if ige1
dladm create-aggr -P L2 -l ige0 -l ige1 aggr1
ipadm create-addr -T dhcp aggr1/v4

should work for you. This is assuming you have disabled the nwam service and are using the physical service:

svcadm disable svc:/network/physical:nwam
svcadm enable svc:/network/physical:default

On disk failures:

So if you have a bunch of disks in a ZFS pool how can you move them to a new server? If you have done it like suggested (Use HBA controller, never use hardware-raid) you can just plug your disks into your new computer with any disk controller and import your pool - no problem -

Quote: How does rebuilding work when you lose a drive? If you have set ZFS pool property autoreplace=on you just need to replace a failed drive, otherwise plug in a new disk and do a replace failled drive → new disk

If your controller does not support hot-plug you need a reboot after plug-in new disks

Gea

On SMB file sharing:

Hello, I run Solaris 11 Express and napp-it for stuff liek SMB and zpools

I was wondering how to I make custom SMB users or groups, like users that only can access some files, they can read at one place and write on another, how does this work? With napp-it you can create user and smb-groups in menu user. Connect from Windows as user root and set desired file and folder ACL (works from Win XP pro, Win 2003 and Win 7 pro, problems are reported with home editions and Win 7 ultimate)

Problem: Solaris ACL are order sensitive - Windows ACL not non-trivial ACL should be set from Solaris

From Solaris you can set ACL via CLI or via napp-it ACL extension (in development, currently you can set share level ACL and ACL on shared folders not on other files and folders)

Gea

Exactly what Gea said, but to give you specific commands

#zpool import to list all the pools found

#zpool import poolname to import the pool.

If you had a failure and didn't do a clean export you just need to use the -f (force) switch #zpool import -f poolname

About overflow protection:

Its always bad, if you fill up your pool up to 100% with data. Overflow protection sets a 10% reservation on the pool ZFS itself.

The available space for your ZFS folder is then 90%. That means, if you fill your folder up to the max, 10% of the pool remains always free. You can check/modify that always in menu ZFS folder under reservation.

Gea