Virtual Architects Support Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== OpenIndiana ======

See also **[[va:va_shared_storage|Shared Storage System Notes]]**.

See also **[[computing:storage:napp-it|napp-it]]**.

See also **[[computing:storage:zfs|ZFS]]**.

:!: For large-memory systems, choose a larger boot drive because the swap and dump devices take up a lot of space by default.

===== Documentation =====

http://technodrone.blogspot.com/2012/05/openindiana-installation-walkthrough.html

http://wiki.openindiana.org/oi/Using+OpenIndiana

==== Solaris 11 Cheat Sheets ====

  * How to create network configuration profiles?
  * How to configure VLANS?
  * How to manipulate Zones?
  * How to use ZFS shadow migration?

Solaris 11 Administrator Cheat Sheet: http://www.oracle.com/technetwork/server-storage/solaris11/documentation/solaris-11-cheat-sheet-1556378.pdf

Solaris 11 Package Manager Cheat Sheet: http://www.oracle.com/technetwork/server-storage/solaris11/documentation/ips-one-liners-032011-337775.pdf

===== Full Reboot =====

A normal 'reboot' command does a fast reboot.  This does a full reboot:

<file>
reboot -p
</file>

===== Installing =====

Download the CD image and burn it to a disc, or download the USB image and the 1G.header files, then:

<file>
cat /path/to/1G.header /path/to/oi-dev-151a7-text-x86.usb | dd bs=1024k of=/dev/sdx
</file>

===== Permit Root Login =====

http://blog.allanglesit.com/2012/07/solaris-11-permit-root-logins-locally-and-via-ssh/

===== Managing Dump and Swap =====

Planning for Swap Space: http://docs.oracle.com/cd/E19963-01/html/821-1459/fsswap-31050.html

Adjusting the Sizes of Your ZFS Swap and Dump Devices: http://docs.oracle.com/cd/E19963-01/html/821-1448/ggrln.html

==== Dump ====

:!: You only need a dump device if you need (and usually buy) help from OS devs, example from OmniTi.
If so, you may need a larger rpool commensurate with your server's RAM size.

Estimate needed dump space (128GB RAM):

<file>
root@stor-01:/root# dumpadm -e
Estimated dump size: 61.5G
</file>

Query the current dump device size:

<file>
root@stor-01:/root# zfs get volsize rpool/dump
NAME        PROPERTY  VALUE    SOURCE
rpool/dump  volsize   17.7G    local
</file>

Change the dump volume size:

<file>
zfs get volsize rpool/dump

zfs set volsize=4G rpool/dump
</file>

Disable dump and delete the dump volume:

<file>
dumpadm -d none
zfs destroy rpool/dump
</file>

==== Swap ====

:!: You can probably skip this section if you are configuring a storage server as the swap space is not pre-allocated.

Note the change in blocks allocated to swap:

<file>
swap -lh
swap -d /dev/zvol/dsk/rpool/swap
zfs volsize=4G rpool/swap
swap -a /dev/zvol/dsk/rpool/swap
swap -lh
</file>

===== Package Management =====

http://docs.oracle.com/cd/E19963-01/html/820-6572/managepkgs.html

<file>
pkg publisher

pkg unset-publisher opensolaris.org
</file>

In addition to the base Operating System repositories, we also add some extra repos:

<file>
pkg set-publisher -p http://pkg.openindiana.org/sfe
pkg set-publisher -p http://pkg.openindiana.org/sfe-encumbered
</file>

===== Upgrading OpenIndiana =====

http://wiki.openindiana.org/oi/Upgrading+OpenIndiana

Update the 'pkg' package manager:

<file>
pkg install package/pkg
</file>

You can then do a trial run image update:

<file>
pkg image-update -nv
</file>

Run it again without the 'n' flag to actually do the update. This will create a new boot environment.

===== Manage Boot Environments =====

http://docs.oracle.com/cd/E23824_01/html/E21801/administer.html

<file>
beadm list                                    # list BE's
beadm rename <be-name> <new-be-name>          # rename a BE - can't rename active BE
beadm destroy <be-name>                       # delete a BE
beadm activate <be-name>                      # make another BE active on reboot
</file>

===== Update OpenIndiana =====

The 'n' switch is for trial runs.  Omit the 'n' to update.

<file>
pkg install package/pkg

pkg update -vn
</file>

===== User Management =====

:!: By default, you must log into OI as a regular user before using 'su' to gain 'root' access.

==== Add User ====

<file>
useradd -g staff -d /home/newuser -m -s /bin/bash -c "Full Name" newuser
</file>

Make 'newuser' an administrator (sudo privileges):

<file>
visudo
</file>

Check for errors:

<file>
pwck
</file>

==== User Roles (RBAC) ====

https://blogs.oracle.com/observatory/en_US/entry/understading_rbac

Allow 'newuser' to 'su' to 'root':

<file>
usermod -R root newuser
</file>

===== Networking =====

:!: Solaris 11 networking is significantly different from previous versions.

New tools supersede the older ''ipconfig'' and ''ndd'' tools:

  * ''dladm''
  * ''ipadm''
  * Changes persist after reboot

Solaris 11 also supports network configuration profiles:

  * The ''Automatic'' NCP uses DHCP
  * The ''DefaultFixed'' NCP requires the network interfaces to be manually configured
  * New tools:
    * 

**Getting Started Guide for Solaris 11**: 

http://www.oracle.com/technetwork/articles/servers-storage-admin/s11-network-config-1632927.html

http://wiki.openindiana.org/oi/Static+IP

http://www.rainingpackets.com/configuring-static-ip-openindiana/

http://docs.oracle.com/cd/E19963-01/html/821-1458/gjwiq.html#gjgob%23scrolltoc

http://www.linuxquestions.org/questions/solaris-opensolaris-20/how-to-configure-network-with-solaris-221164/

<file>
dladm show-phys

dladm show-link

ipadm show-if [interface]

ipadm show-addr [addrobj]

ipadm create-addr -T address-type -a address/prefixlen addrobj

ipadm show-addrprop [-p property] [addrobj]
</file>

==== Intel x540 10GbE NIC ====

:!: It seems the latest OI now has this driver and these steps are no longer necessary.

Determine the 'vendor' and 'device' id's that aren't being recognized by the 'ixgbe' driver, then add them to the list:

<file>
scanpci

grep ixgbe /etc/driver_aliases

update_drv -a -i "pci8086,1528" ixgbe

ifconfig -a plumb
ifconfig -a

modinfo|grep ixgbe

more /etc/path_to_inst | grep ixgbe
</file>

==== Add New Persistent NIC ====

:!: If you are using Napp-IT, use it for assigning IP addresses.

<file>
vi /etc/hostname.ixgbe0
</file>

<file>
10.1.10.5 netmask 255.255.255.0
</file>

<file>
vi /etc/hostname.ixgbe1
</file>

<file>
10.1.10.6 netmask 255.255.255.0
</file>

<file>
vi /etc/inet/hosts
</file>

<file>
10.1.10.5       10g-eth-1 
10.1.10.6       10g-eth-2
</file>

==== Configure Jumbo Frames ====

=== Intel X540 10GbE NIC ===

http://docs.oracle.com/cd/E19254-01/820-7895-11/cggfhffg.html

<file>
nano /kernel/drv/ixgbe.conf

default_mtu             = 9000;

reboot
</file>

=== General Jumbo Frames Info ===

http://docs.oracle.com/cd/E19862-01/820-1606-12/jumbo_fms.html

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2012445

https://blogs.oracle.com/taylor22/entry/nfs_streaming_over_10_gbe

http://docs.oracle.com/cd/E19254-01/820-7895-11/cggfhffg.html

http://hardforum.com/showthread.php?t=1601960

  * For optimal network speed, use high quality NICs such as Intel server NICs
  * Jumbo frames helps for large data transfers, but it is slower for small data
    * For example a database that read/writes less than 8k chunks
  * All devices using jumbo frames should run in their own VLAN
  * Set the MTU to 9000

=== Configure Jumbo Frames on OpenIndiana ===
 
Your NIC may be identified as ''ixgbe0''. If your identifier differs, substitute it appropriately in these commands.

Enable Jumbo Frames using this command:

<file>
ndd -set /dev/ixgbe0 accept-jumbo 1
</file>

Set the MTU on the adapter to 9000 using this command:

<file>
ifconfig ixgbe0 mtu 9000
</file>

<file>
ipadm set-ifprop mtu 9000
</file>

=== Test Jumbo Frames ===

You can test the settings by pinging to and from the machine with 9000 byte packets.

To test from a remote client, run this command:

<file>
ping -s 9000 IP_of_Solaris_machine
</file>

To test from the Solaris machine, run this command:

<file>
ping -s IP_of_another_machine 9000 4
</file>

===== Services (Daemons) =====

List services:

<file>
svcs
</file>

After making configuration file changes:

<file>
svcadm refresh <servicename>
</file>

:!: If services show as 'maintenance' rather than 'online', there is probably a configuration error.

===== NTP =====

Configure NTP:

<file>
cp /etc/inet/ntp.client /etc/inet/ntp.conf

vi /etc/inet/ntp.conf
</file>

<file>
server 0.us.pool.ntp.org iburst
server 1.us.pool.ntp.org
server 2.us.pool.ntp.org
server 3.us.pool.ntp.org
</file>

Set the time and enable the NTP daemon:

<file>
ntpdate pool.ntp.org

svcadm enable ntp
</file>

NTP takes a while to get synchronized.  Check the status (on localhost) like this:

<file>
ntpq -p
</file>

===== Mirror the Boot Drive (on x86 System) =====

http://wiki.openindiana.org/oi/2.1+Post-installation

http://darkstar-solaris.blogspot.co.uk/2008/09/zfs-root-mirror.html

List all disks and ZFS pools:

<file>
format
<ctrl-C>

cfgadm -s "select=type(disk)"

zpool status rpool
</file>

In the following example:

  * c5t2d0 is the initial installation drive
  * c5t3d0 is the new disk we want to become the mirror
  * p0 is entire disk

Create one large partition:

<file>
pfexec fdisk -B c5t3d0p0
</file>

Set up SMI label with same partitioning as original boot disk:

<file>
pfexec prtvtoc /dev/rdsk/c5t2d0s2 | pfexec fmthard -s - /dev/rdsk/c5t3d0s2
</file>

Add the second drive to the rpool:

<file>
pfexec zpool attach -f rpool c5t2d0s0 c5t3d0s0
</file>

:!: Do not continue until the resilver is finished.

Check resilver progress:

<file>
zpool status -v rpool
</file>

Make the mirror drive bootable:

<file>
pfexec installgrub /boot/grub/stage1 /boot/grub/stage2 /dev/rdsk/c5t3d0s0
</file>

:!: To boot to the second drive, select it in the BIOS as the first boot device.

<file>
reboot -p
</file>

====== Notes from Others ======

===== Shutdown / Reboot Commands =====

  * Poweroff
  * Reboot

===== Allow Root SSH =====

  - vi /etc/ssh/sshd_config
    - change permitrootlogin to yes
  - vi /etc/default/login
    - change line: CONSOLE =/dev/login|console (Comment it out may be better)
  - usermod -K type=normal root
  - reboot

===== Steps for Solaris Raid1 =====

  - Create Solaris parition with format/fdisk on second disk (c3t10d0)
  - copy volumes from disk 0 to disk 1
    - prtvtoc /dev/rdsk/c3t0d0s0 | fmthard -s - /dev/rdsk/c3t1d0s0
  - Add second drive to rpool by attaching drive 0 and 1
    - Zpool attach -f rpool c3t0d0s0 c3t1d0s0
  - Check status: zpool status rpool

===== Steps for IP Address Configuration =====

  - If no ip addresses exist in ifconfig -a edit /etc/hosts and make sure you have an entry for static ip and hostname.
  - svcadm disable svc:/network/physical:nwam
  - svcadm enable svc:/network/physical:default
  - dladm show-phys

==== Optional - Rename Link ====

  - dladm rename-link oldname Net.1
  - ipadm create-addr -T static -a 192.168.10.7/24 Net.1/v4
    - route -p add default 192.168.10.254
  - check for dns server info:
    - cat /etc/resolv.conf
    - add our dns with vi if its missing: nameserver  192.168.10.10
  - Check that we are configured to use files and dns for resolution:
    - cat /etc/nsswitch.conf
    - vi  /etc/nsswitch.conf
      - put files dns for hosts, ipnodes, and networks.

===== Enable Jumbo Frames =====

<file>
dladm set-linkprop -p mtu=9000 InterfaceName
</file>

===== Link Aggregation =====

Similar terms: NIC Bonding, NIC Teaming

<file>
dladm show-link
dladm create-aggr -l Net.LACP.1 -l Net.LACP.2 Net.Team.1
ipadm create-addr -T static -a 192.168.100.7/24 Net.Team.1
dladm modify-aggr -L active -T short Net.Team.1
dladm modify-aggr -P L2 Net.Team.1
</file>

:!: Configure cisco to use mode "active"
 
===== Steps for Solaris Raid 10 =====

<file>
zpool create tank mirror disk1 disk2
zpool add tank mirror disk3 disk4
zpool add tank mirror disk5 disk6 #etc...
</file>

===== Steps for Adding Cache Drives =====

<file>
zpool add tank cache disk1
zpool add tank cache disk2
</file>

===== Disable ZIL =====

<file>
zfs set sync=disabled tank
zfs get sync
</file>

===== Steps for Setting Up Comstar =====

<file>
svcadm enable svc:/system/stmf:default
zpool list #will tell you capacity, substitute capacity in next line, its currently 20G
zfs create -V 20G tank/ISCSI.tank
sbdadm create-lu /dev/zvol/rdsk/tank/ISCSI.tank
sbdadm list-lu
stmfadm add-view #LUN Name/Number
pkg install -v SUNWiscsit
svcadm enable -r svc:/network/iscsi/target:default
itadm create-target
</file>

===== Install GUI (For Management/Reporting if you care) =====

<file>
wget -O - www.napp-it.org/nappit | perl
</file>

===== Update Solaris =====

  - Check what updates are available: <file>pfexec pkg image-update -nv</file>
  - apply updates if any are available: <file>pfexec pkg image-update -v</file>
Virtual Architects Support Wiki

User Tools

Site Tools

Page Tools
