What does systemctl do?
In my earlier post on nginx and uWSGI, I used systemctl to kick off my app. I admittedly don't know much about systemctl (or about CentOS, if we're being totally honest), so I wanted to dig more into that.
My first round of looking for documentation took me to the Fedora project's documentation on services and daemons. (Turns out that CentOS is ultimately a fork of Fedora) The official documentation seems like a good place to start here. It gives a nice description of how to check the status of my service, so I'm going to do that here.
$ systemctl status dashboard.service dashboard.service - uwsgi instance to serve dashboard Loaded: loaded (/etc/systemd/system/dashboard.service; enabled) Active: active (running) since Sat 2015-10-17 04:32:57 UTC; 10h ago Main PID: 14914 (uwsgi) CGroup: /system.slice/dashboard.service ├─14914 /var/www/dashboard/venv/bin/uwsgi --ini dashboard.ini ├─14916 /var/www/dashboard/venv/bin/uwsgi --ini dashboard.ini ├─14917 /var/www/dashboard/venv/bin/uwsgi --ini dashboard.ini ├─14918 /var/www/dashboard/venv/bin/uwsgi --ini dashboard.ini ├─14919 /var/www/dashboard/venv/bin/uwsgi --ini dashboard.ini └─14920 /var/www/dashboard/venv/bin/uwsgi --ini dashboard.ini
Oct 17 04:32:57 [INFO ABOUT MY LINODE] uwsgi[14914]: spawned uWSGI worker 5 (pid: 14920, cores: 1) Oct 17 04:33:00 [INFO ABOUT MY LINODE] uwsgi[14914]: [pid: 14920|app: 0|req: 1/1] [WHAT LOOKS LIKE AN IP ADDRESS] () {42 vars in 722 b...re 0) Oct 17 04:33:00 [INFO ABOUT MY LINODE] uwsgi[14914]: [pid: 14919|app: 0|req: 1/2] [WHAT LOOKS LIKE AN IP ADDRESS] () {42 vars in 713 b...re 0) Oct 17 04:33:00 [INFO ABOUT MY LINODE] uwsgi[14914]: [pid: 14920|app: 0|req: 2/3] [WHAT LOOKS LIKE AN IP ADDRESS] () {42 vars in 680 b...re 0) Oct 17 04:35:56 [INFO ABOUT MY LINODE] uwsgi[14914]: [pid: 14920|app: 0|req: 3/4] [WHAT LOOKS LIKE AN IP ADDRESS] () {42 vars in 722 b...re 0) Oct 17 04:35:56 [INFO ABOUT MY LINODE] uwsgi[14914]: [pid: 14916|app: 0|req: 1/5] [WHAT LOOKS LIKE AN IP ADDRESS] () {42 vars in 713 b...re 0) Oct 17 04:35:59 [INFO ABOUT MY LINODE] uwsgi[14914]: [pid: 14920|app: 0|req: 4/6] [WHAT LOOKS LIKE AN IP ADDRESS] () {42 vars in 680 b...re 0) Oct 17 04:37:15 [INFO ABOUT MY LINODE] uwsgi[14914]: [pid: 14920|app: 0|req: 5/7] [WHAT LOOKS LIKE AN IP ADDRESS] () {42 vars in 722 b...re 0) Oct 17 04:37:16 [INFO ABOUT MY LINODE] uwsgi[14914]: [pid: 14919|app: 0|req: 2/8] [WHAT LOOKS LIKE AN IP ADDRESS] () {42 vars in 713 b...re 0) Oct 17 04:37:16 [INFO ABOUT MY LINODE] uwsgi[14914]: [pid: 14916|app: 0|req: 2/9] [WHAT LOOKS LIKE AN IP ADDRESS] () {42 vars in 680 b...re 0) Hint: Some lines were ellipsized, use -l to show in full.
I masked out what I think would be identifying details that I don't want out there. The first commented-out section was my Linode's name, but the second was an IP address that I did not recognize. Maybe it's my IPv4 address? Maybe it's the Linode's hypervisor IP address? I think, though, that these logs may be from uWSGI and not from systemctl. I say this because the Fedora documentation doesn't have anything that looks like this in their page about these logs. I'm going to skip it for now. Most of the other information on Fedora's page here is about using the tool, which I've already done, so I'm going to move on to the raw man pages.
The man page for systemctl gives the following description for its use:
systemctl — Control the systemd system and service manager
Okay, so systemctl is actually used to control the systemd tool. Maybe the documentation was a little sparse because systemd is where the money is. Still, let's go through the man page to see what we can get here. I'm going to punch through the commands one by one to walk through what they can do.
$ systemctl -t systemctl: option requires an argument -- 't'
Mer, that didn't go well. I didn't read the documentation enough. It requires a listing of unit types, which I can find here:
$ systemctl -t help Available unit types: service socket target device mount automount snapshot timer swap path slice scope
$ systemctl -t service
UNIT LOAD ACTIVE SUB DESCRIPTION .... stuff ... crond.service loaded active running Command Scheduler dashboard.service loaded active running uwsgi instance to serve dashboard dbus.service loaded active running D-Bus System Message Bus .... stuff ...
Aha! There's my dashboard service, sandwiched right in between cron and something called dbus. It looks like it's loaded, active, and running, which is what I would expect. $ systemctl --state=active provides the same information, just a different way to cut the data.
$ systemctl -t service --recursive
This doesn't work as expected on my machine. It may be live on Fedora, but disabled on CentOS or something like that.
$ systemctl -t service --reverse
The man page says this should do the following: "Show reverse dependencies between units with list-dependencies, i.e. follow dependencies of type WantedBy=, RequiredBy=, RequiredByOverridable=, PartOf=, BoundBy=, instead of Wants= and similar." I remember having to type WantedBy= in my configuration earlier:
$ vi /etc/systemd/system/dashboard.service
[Unit] Description=uwsgi instance to serve dashboard After=network.target
[Service] User=ghost Group=nginx WorkingDirectory=/var/www/dashboard Environment="PATH=/var/www/dashboard/venv/bin" ExecStart=/var/www/dashboard/venv/bin/uwsgi --ini dashboard.ini
[Install] WantedBy=multi-user.target
Okay, so I must have configured dashboard.service to start with some sort of a dependency: this multi-user.target thing. Let's see if I can find that.
$ systemctl -t target
UNIT LOAD ACTIVE SUB DESCRIPTION ... stuff ... local-fs.target loaded active active Local File Systems multi-user.target loaded active active Multi-User System network.target loaded active active Network ... stuff ...
It looks like these "targets" are super important. I didn't know you could even have a Unix system without multiple users, but it looks like that's something you have to manually turn on - and can turn off. Crazy.
There are two commands here that I don't want to fiddle with too much for now, because I plan to go down a systemd rabbit hole and these look like I could cause problems if I don't know what I'm doing. I'll circle back to them.
$ systemctl --job-mode= ? $ systemctl --fail
The rest of the options here look like they're tied to actually starting and stopping services, rather than just managing them. Rather than try to imply what's going on with these, I want to jump down to the Commands section so I can start understanding what this utility can do.
$ systemctl list-units
This does the same thing as just typing systemctl, and it dumps out an unfiltered list of running services and sockets and things.
$ systemctl start [SOMETHING] $ systemctl stop [SOMETHING] $ systemctl restart [SOMETHING]
I used these last night, and they do exactly what I'd expect. They start, stop, and restart services. This one's new though:
$ systemctl reload [SOMETHING]
When we type this, we can reload the configuration of a service without actually reloading the service itself. We could reload nginx.conf by typing systemctl reload nginx.
Now for what is the unexpectedly big kahuna:
$ systemctl show dashboard
Id=dashboard.service Names=dashboard.service Requires=basic.target Wants=system.slice WantedBy=multi-user.target Conflicts=shutdown.target Before=shutdown.target multi-user.target After=network.target systemd-journald.socket basic.target system.slice Description=uwsgi instance to serve dashboard LoadState=loaded ActiveState=active SubState=running FragmentPath=/etc/systemd/system/dashboard.service UnitFileState=enabled InactiveExitTimestamp=Sat 2015-10-17 04:32:57 UTC InactiveExitTimestampMonotonic=13928551290 ActiveEnterTimestamp=Sat 2015-10-17 04:32:57 UTC ActiveEnterTimestampMonotonic=13928551290 ActiveExitTimestamp=Sat 2015-10-17 04:31:52 UTC ActiveExitTimestampMonotonic=13863477373 InactiveEnterTimestamp=Sat 2015-10-17 04:31:52 UTC InactiveEnterTimestampMonotonic=13863477373 CanStart=yes CanStop=yes CanReload=no CanIsolate=no StopWhenUnneeded=no RefuseManualStart=no RefuseManualStop=no AllowIsolate=no DefaultDependencies=yes OnFailureIsolate=no IgnoreOnIsolate=no IgnoreOnSnapshot=no NeedDaemonReload=no JobTimeoutUSec=0 ConditionTimestamp=Sat 2015-10-17 04:32:57 UTC ConditionTimestampMonotonic=13928549499 ConditionResult=yes Transient=no Slice=system.slice ControlGroup=/system.slice/dashboard.service Type=simple Restart=no NotifyAccess=none RestartUSec=100ms TimeoutStartUSec=1min 30s TimeoutStopUSec=1min 30s WatchdogUSec=0 WatchdogTimestampMonotonic=0 StartLimitInterval=10000000 StartLimitBurst=5 StartLimitAction=none ExecStart={ path=/var/www/dashboard/venv/bin/uwsgi ; argv[]=/var/www/dashboard/venv/bin/uwsgi --ini dashboard.ini ; ignore_errors=no ; start_time=[Sat 2 PermissionsStartOnly=no RootDirectoryStartOnly=no RemainAfterExit=no GuessMainPID=yes MainPID=14914 ControlPID=0 Result=success Environment=PATH=/var/www/dashboard/venv/bin UMask=0022 LimitCPU=18446744073709551615 LimitFSIZE=18446744073709551615 LimitDATA=18446744073709551615 LimitSTACK=18446744073709551615 LimitCORE=18446744073709551615 LimitRSS=18446744073709551615 LimitNOFILE=4096 LimitAS=18446744073709551615 LimitNPROC=3934 LimitMEMLOCK=65536 LimitLOCKS=18446744073709551615 LimitSIGPENDING=3934 LimitMSGQUEUE=819200 LimitNICE=0 LimitRTPRIO=0 LimitRTTIME=18446744073709551615 WorkingDirectory=/var/www/dashboard OOMScoreAdjust=0 Nice=0 IOScheduling=0 CPUSchedulingPolicy=0 CPUSchedulingPriority=0 TimerSlackNSec=50000 CPUSchedulingResetOnFork=no NonBlocking=no StandardInput=null StandardOutput=journal StandardError=inherit TTYReset=no TTYVHangup=no TTYVTDisallocate=no SyslogPriority=30 SyslogLevelPrefix=yes SecureBits=0 CapabilityBoundingSet=18446744073709551615 User=ghost Group=nginx MountFlags=0 PrivateTmp=no PrivateNetwork=no SameProcessGroup=no IgnoreSIGPIPE=yes NoNewPrivileges=no KillMode=control-group KillSignal=15 SendSIGKILL=yes SendSIGHUP=no CPUAccounting=no CPUShares=1024 BlockIOAccounting=no BlockIOWeight=1000 MemoryAccounting=no MemoryLimit=18446744073709551615 DevicePolicy=auto ExecMainStartTimestamp=Sat 2015-10-17 04:32:57 UTC ExecMainStartTimestampMonotonic=13928551199 ExecMainExitTimestampMonotonic=0 ExecMainPID=14914 ExecMainCode=0 ExecMainStatus=0
I KNOW WHAT LIKE A TENTH OF THIS MEANS. THIS IS AWESOME.
I'm going to take a break here and digest some of the information gleaned from this little exercise. More to come!